Image processing apparatus and method, and image pickup apparatus

ABSTRACT

The present invention aims to recognize the mixture state of images. An area specifying unit  103  specifies a mixed area in which foreground object components which form a foreground object and background object components which form a background object are mixed. An estimated-mixture-ratio calculator  104  detects, in correspondence with the image data, the mixture ratio indicating the ratio of the mixture of the foreground object components to the mixture of the background object components in the mixed area in which the foreground object components and the background object components are mixed. At least one of the area specifying unit  103  and the estimated-mixture-ratio calculator  104  performs image processing on the basis of a plurality of types of components. The present invention can be applied to an image processing apparatus.

TECHNICAL FIELD

[0001] The present invention relates to image processing apparatuses andmethods, and image-capturing apparatuses, and more particularly, to animage processing apparatus and method, and an image-capturing apparatusin which a difference between a signal detected by a sensor and the realworld is taken into consideration.

BACKGROUND ART

[0002] A technique for detecting incidents occurring in the real worldby a sensor and for processing sampled data output from the image sensoris widely used.

[0003] For example, motion blur occurs in an image obtained by capturingan object moving in front of a predetermined stationary background witha video camera if the moving speed is relatively high.

[0004] However, when an object is moving in front of a stationarybackground, not only does motion blur caused by the mixture of themoving object itself occur, but also the mixture of the background imageand the object image occurs. Hitherto, processing corresponding to themixture state of the background image and the moving object has not beenconsidered.

DISCLOSURE INVENTION

[0005] The present invention has been made in view of suchcircumstances. Accordingly, it is an object of the present invention toknow the mixture state of images.

[0006] A first image processing apparatus of the present inventioncomprises area specifying means for specifying, in correspondence withimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed; and mixture-ratio detection means fordetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed, wherein at least one of the area specifying meansand the mixture ratio detection means performs image processing on thebasis of the plurality of types of components.

[0007] The area specifying means may comprise component mixed-areadetection means for detecting the mixed area for each of the pluralityof types of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and mixed-area specifying means for specifying the mixedarea corresponding to the image data on the basis of the detectionresult of the mixed area corresponding to the plurality of types ofcomponents detected by the component mixed-area detection means.

[0008] The area specifying means may comprise space-correlation-valuecalculation means for calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of the image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of the designated pixelin the spatial direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; time-correlation-valuecalculation means for calculating a time correlation value indicating acorrelation between the designated pixel data and pixel data of a timeneighboring pixel positioned in the neighborhood of the designated pixelin the time direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; and foreground areaspecifying means for specifying a foreground area formed of only theforeground object components on the basis of the space correlation valueand the time correlation value corresponding to the designated pixel.

[0009] The area specifying means may comprise mixed-area specifyingmeans for specifying the mixed area on the basis of the foreground areaof the designated frame and the foreground area of a neighboring framein the neighborhood of the designated frame.

[0010] The mixture-ratio detection means may comprise componentmixture-ratio detection means for detecting the mixture ratio for eachof the plurality of types of components; and component integratedmixture-ratio detection means for detecting the mixture ratiocorresponding to the image data by integrating the detection results ofthe mixture ratios corresponding to the plurality of types of componentsdetected by the component mixture-ratio detection means.

[0011] The mixture-ratio detection means may comprise integration meansfor integrating the pixel values of the plurality of types of componentsfor each pixel and for outputting the value as integrated data; andintegrated data mixture-ratio detection means for detecting the mixtureratio corresponding to the image data on the basis of the integrateddata.

[0012] The integration means may add the pixel values of the pluralityof types of components for each pixel and may output the added result asthe integrated data.

[0013] An image processing method of the present invention comprises anarea specifying step of specifying, in correspondence with image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed; a mixture-ratio detection step of detecting, incorrespondence with the image data, the mixture ratio indicating theratio of the mixture of the foreground object components to the mixtureof the background object components in a mixed area in which theforeground object components and the background object components aremixed; and an output control step of controlling the output of thedetected mixture ratio, wherein at least one of the area specifying stepand the mixture-ratio detection step performs image processing on thebasis of the plurality of types of components.

[0014] The area specifying step may comprise a component mixed-areadetection step of detecting the mixed area for each of the plurality oftypes of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and a mixed-area specifying step of specifying the mixedarea corresponding to the image data on the basis of the detectionresult of the mixed area corresponding to the plurality of types ofcomponents detected in the component mixed-area detection step.

[0015] The area specifying step may comprise a space-correlation-valuecalculation step of calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of the image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of the designated pixelin the spatial direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; atime-correlation-value calculation step of calculating a timecorrelation value indicating a correlation between the designated pixeldata and pixel data of a time neighboring pixel positioned in theneighborhood of the designated pixel in the time direction on the basisof the plurality of types of components corresponding to the designatedpixel; and a foreground area specifying step of specifying a foregroundarea formed of only the foreground object components on the basis of thespace correlation value and the time correlation value corresponding tothe designated pixel.

[0016] The area specifying step may comprise a mixed-area specifyingstep of specifying the mixed area on the basis of the foreground area ofthe designated frame and the foreground area of a neighboring frame inthe neighborhood of the designated frame.

[0017] The mixture-ratio detection step may comprise a componentmixture-ratio detection step of detecting the mixture ratio for each ofthe plurality of types of components; and a component integratedmixture-ratio detection step of detecting the mixture ratiocorresponding to the image data by integrating the detection results ofthe mixture ratios corresponding to the plurality of types of componentsdetected in the component mixture-ratio detection step.

[0018] The mixture-ratio detection step may comprise an integration stepof integrating the pixel values of the plurality of types of componentsfor each pixel and for outputting the value as integrated data; and adata mixture-ratio detection step of detecting the mixture ratiocorresponding to the image data on the basis of the integrated data.

[0019] In the integration step, the pixel values of the plurality oftypes of components may be added for each pixel, and the added resultmay be output as the integrated data.

[0020] A program of a first recording medium of the present inventioncomprises an area specifying step of specifying, in correspondence withimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed; a mixture-ratio detection step ofdetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed; and an output control step of controlling theoutput of the detected mixture ratio, wherein at least one of the areaspecifying step and the mixture-ratio detection step performs imageprocessing on the basis of the plurality of types of components.

[0021] The area specifying step may comprise a component mixed-areadetection step of detecting the mixed area for each of the plurality oftypes of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and a mixed-area specifying step of specifying the mixedarea corresponding to the image data on the basis of the detectionresult of the mixed area corresponding to the plurality of types ofcomponents detected in the component mixed-area detection step.

[0022] The area specifying step may comprise a space-correlation-valuecalculation step of calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of the image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of the designated pixelin the spatial direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; atime-correlation-value calculation step of calculating a timecorrelation value indicating a correlation between the designated pixeldata and pixel data of a time neighboring pixel positioned in theneighborhood of the designated pixel in the time direction on the basisof the plurality of types of components corresponding to the designatedpixel; and a foreground area specifying step of specifying a foregroundarea formed of only the foreground object components on the basis of thespace correlation value and the time correlation value corresponding tothe designated pixel.

[0023] The area specifying step may comprise a mixed-area specifyingstep of specifying the mixed area on the basis of the foreground area ofthe designated frame and the foreground area of a neighboring frame inthe neighborhood of the designated frame.

[0024] The mixture-ratio detection step may comprise a componentmixture-ratio detection step of detecting the mixture ratio for each ofthe plurality of types of components; and a component integratedmixture-ratio detection step of detecting the mixture ratiocorresponding to the image data by integrating the detection resultscorresponding to the plurality of types of components detected in thecomponent mixture-ratio detection step.

[0025] The mixture-ratio detection step may comprise an integration stepof integrating the pixel values of the plurality of types of componentsfor each pixel and outputting the value as integrated data; and anintegrated data mixture-ratio detection step of detecting the mixtureratio corresponding to the image data on the basis of the integrateddata.

[0026] In the integration step, the pixel values of the plurality oftypes of components may be added for each pixel, and the added resultmay be output as the integrated data.

[0027] A first program of the present invention causes a computer toexecute an area specifying step of specifying, in correspondence withimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed; a mixture-ratio detection step ofdetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object in a mixed area in which theforeground object components and the background object components aremixed; and an output control step of controlling the output of thedetected mixture ratio, wherein, in at least one of the area specifyingstep and the mixture-ratio detection step, image processing is performedon the basis of the plurality of types of components.

[0028] The area specifying step may comprise a component mixed-areadetection step of detecting the mixed area for each of the plurality oftypes of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and a mixed-area specifying step of specifying the mixedarea corresponding to the image data on the basis of the detectionresult of the mixed area corresponding to the plurality of types ofcomponents detected in the component mixed-area detection step.

[0029] The area specifying step may comprise a space-correlation-valuecalculation step of calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of the image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of the designated pixelin the spatial direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; atime-correlation-value calculation step of calculating a timecorrelation value indicating a correlation between the designated pixeldata and pixel data of a time neighboring pixel positioned in theneighborhood of the designated pixel in the time direction on the basisof the plurality of types of components corresponding to the designatedpixel; and a foreground area specifying step of specifying a foregroundarea formed of only the foreground object components on the basis of thespace correlation value and the time correlation value corresponding tothe designated pixel.

[0030] The area specifying step may comprise a mixed-area specifyingstep of specifying the mixed area on the basis of the foreground area ofthe designated frame and the foreground area of a neighboring frame inthe neighborhood of the designated frame.

[0031] The mixture-ratio detection step may comprise a componentmixture-ratio detection step of detecting the mixture ratio for each ofthe plurality of types of components; and a component integratedmixture-ratio detection step of detecting the mixture ratiocorresponding to the image data by integrating the detection results ofthe mixture ratios corresponding to the plurality of types of componentsdetected in the component mixture-ratio detection step.

[0032] The mixture-ratio detection step may comprise an integration stepof integrating the pixel values of the plurality of types of componentsfor each pixel and for outputting the value as integrated data; and anintegrated data mixture-ratio detection step of detecting the mixtureratio corresponding to the image data on the basis of the integrateddata.

[0033] In the integration step, the pixel values of the plurality oftypes of components may be added for each pixel, and the added resultmay be output as the integrated data.

[0034] A first image-capturing apparatus of the present inventioncomprises image-capturing means for outputting a subject image capturedby an image-capturing device including a predetermined number of pixels,the pixels having a time integrating function, as image data which isformed of a predetermined number of pixel data having a plurality oftypes of components at the same pixel position; area specifying meansfor specifying, in correspondence with image data, a mixed area in whichforeground object components which form a foreground object andbackground object components which form a background object are mixed;and mixture-ratio detection means for detecting, in correspondence withthe image data, the mixture ratio indicating the ratio of the mixture ofthe foreground object components to the mixture of the background objectcomponents in a mixed area in which the foreground object components andthe background object components are mixed, wherein at least one of thearea specifying means and the mixture-ratio detection means performsimage processing on the basis of the plurality of types of components.

[0035] The area specifying means may comprise component mixed-areadetection means for detecting the mixed area for each of the pluralityof types of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and mixed-area specifying means for specifying the mixedarea corresponding to the image data on the basis of the detectionresult of the mixed area corresponding to the plurality of types ofcomponents detected by the component mixed-area detection means.

[0036] The area specifying means may comprise space-correlation-valuecalculation means for calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of the image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of the designated pixelin the spatial direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; time-correlation-valuecalculation means for calculating a time correlation value indicating acorrelation between the designated pixel data and pixel data of a timeneighboring pixel positioned in the neighborhood of the designated pixelin the time direction on the basis of the plurality of types ofcomponents corresponding to the designated pixel; and foreground areaspecifying means for specifying a foreground area formed of only theforeground object components on the basis of the space correlation valueand the time correlation value corresponding to the designated pixel.

[0037] The area specifying means may comprise mixed-area specifyingmeans for specifying the mixed area on the basis of the foreground areaof the designated frame and the foreground area of a neighboring framein the neighborhood of the designated frame.

[0038] The mixture-ratio detection means may comprise componentmixture-ratio detection means for detecting the mixture ratio for eachof the plurality of types of components; and component integratedmixture-ratio detection means for detecting the mixture ratio fordetecting the mixture ratio corresponding to the image data byintegrating the detection results of the mixture ratios corresponding tothe plurality of types of components detected by the componentmixture-ratio detection means.

[0039] The mixture ratio detection means may comprise integration meansfor integrating the pixel values of the plurality of types of componentsfor each pixel and for outputting the value as integrated data; andintegrated data mixture-ratio detection means for detecting the mixtureratio corresponding to the image data on the basis of the integrateddata.

[0040] The integration means may add the pixel values of the pluralityof types of components for each pixel and may output the added result asthe integrated data.

[0041] A second image processing apparatus of the present inventioncomprises image data obtaining means for obtaining image data; andprocessing performing means for performing, on the basis of theplurality of types of components of the obtained image data, one ofprocessings of (i) an area specifying step of specifying, incorrespondence with the image data, a mixed area in which foregroundobject components which form a foreground object and background objectcomponents which form a background object are mixed and (ii) amixture-ratio detection step of detecting, in correspondence with theimage data, the mixture ratio indicating the ratio of the mixture of theforeground object components to the mixture of the background objectcomponents in a mixed area in which the foreground object components andthe background object components are mixed.

[0042] The processing performing means may perform, on the basis of theplurality of types of components of the obtained image data, an areaspecifying step of specifying, in correspondence with the image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed.

[0043] The processing performing means may perform, on the basis of theplurality of types of components of the obtained image data, amixture-ratio detection step of detecting, in correspondence with theimage data, the mixture ratio indicating the ratio of the mixture of theforeground object components to the mixture of the background objectcomponents in a mixed area in which the foreground object components andthe background object components are mixed.

[0044] A second image processing method of the present inventioncomprises an image data obtaining step of obtaining image data; and aprocessing performing step of performing, on the basis of the pluralityof types of components of the obtained image data, one of processings of(i) an area specifying step of specifying, in correspondence with theimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed.

[0045] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, an area specifyingstep of specifying, in correspondence with the image data, a mixed areain which foreground object components which form a foreground object andbackground object components which form a background object are mixedmay be performed.

[0046] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, a mixture-ratiodetection step of detecting, in correspondence with the image data, themixture ratio indicating the ratio of the mixture of the foregroundobject components to the mixture of the background object components ina mixed area in which the foreground object components and thebackground object components are mixed may be performed.

[0047] A program of a second recording medium of the present inventioncomprises an image data obtaining step of obtaining image data; and aprocessing performing step of performing, on the basis of the pluralityof types of components of the obtained image data, one of processings of(i) an area specifying step of specifying, in correspondence with theimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed.

[0048] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, an area specifyingstep of specifying, in correspondence with the image data, a mixed areain which foreground object components which form a foreground object andbackground object components which form a background object are mixedmay be performed.

[0049] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, a mixture-ratiodetection step of detecting, in correspondence with the image data, themixture ratio indicating the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed may be performed.

[0050] A second program of the present invention causes a computer toexecute an image data obtaining step of obtaining image data; and aprocessing performing step of performing, on the basis of the pluralityof types of components of the obtained image data, one of processings of(i) an area specifying step of specifying, in correspondence with theimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with the image data, the mixture ratioindicating the ratio of the mixture of the foreground object componentsto the mixture of the background object components in a mixed area inwhich the foreground object components and the background objectcomponents are mixed.

[0051] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, an area specifyingstep of specifying, in correspondence with the image data, a mixed areain which foreground object components which form a foreground object andbackground object components which form a background object are mixedmay be performed.

[0052] In the processing performing step, on the basis of the pluralityof types of components of the obtained image data, a mixture-ratiodetection step of detecting, in correspondence with the image data, themixture ratio indicating the ratio of the mixture of the foregroundobject components to the mixture of the background object components ina mixed area in which the foreground object components and thebackground object components are mixed may be performed.

[0053] A second image-capturing apparatus of the present inventioncomprises image-capturing means for outputting a subject image capturedby an image-capturing device including a predetermined number of pixels,the pixels having a time integrating function, as image data which isformed of a predetermined number of pixel data having a plurality oftypes of components at the same pixel position; and processingperforming means for performing, on the basis of the plurality of typesof components of the image data, one of processings of (i) an areaspecifying step of specifying, in correspondence with the image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed and (ii) a mixture-ratio detection step of detecting, incorrespondence with the image data, the mixture ratio indicating theratio of the mixture of the foreground object components to the mixtureof the background object components in a mixed area in which theforeground object components and the background object components aremixed.

[0054] The processing performing means may perform, on the basis of theplurality of types of components of the image data, an area specifyingstep of specifying, in correspondence with the image data, a mixed areain which foreground object components which form a foreground object andbackground object components which form a background object are mixed.

[0055] The processing performing means may perform, on the basis of theplurality of types of components of the image data, a mixture-ratiodetection step of detecting, in correspondence with the image data, themixture ratio indicating the ratio of the mixture of the foregroundobject components to the mixture of the background object components ina mixed area in which the foreground object components and thebackground object components are mixed.

[0056] In correspondence with image data, a mixed area in whichforeground object components which form a foreground object andbackground object components which form a background object are mixed isspecified. The mixture ratio indicating the ratio of the mixture of theforeground object components to the mixture of the background objectcomponents in a mixed area in which the foreground object components andthe background object components are mixed is detected. At least one ofthe area specification operation and the mixture-ratio detectionoperation performs image processing on the basis of the plurality oftypes of components.

[0057] Image data is obtained, and based on the plurality of types ofcomponents of the obtained image data, one of processings of (i) an areaspecifying step of specifying, in correspondence with the image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed and (ii) a mixture-ratio detection step of detecting, incorrespondence with the image data, the mixture ratio indicating theratio of the mixture of the foreground object components to the mixtureof the background object components in a mixed area in which theforeground object components and the background object components aremixed is performed.

[0058] As a result, the mixture state of the images can be recognized.

BRIEF DESCRIPTION OF THE DRAWINGS

[0059]FIG. 1 shows an embodiment of an image processing apparatus of thepresent invention.

[0060]FIG. 2 is a block diagram illustrating the image processingapparatus.

[0061]FIG. 3 illustrates the image capturing performed by a sensor.

[0062]FIG. 4 illustrates the arrangement of pixels.

[0063]FIG. 5 illustrates the operation of a detection device.

[0064]FIG. 6A illustrates an image obtained by image-capturing an objectcorresponding to a moving foreground and an object corresponding to astationary background.

[0065]FIG. 6B illustrates a model of an image obtained byimage-capturing an object corresponding to a moving foreground and anobject corresponding to a stationary background.

[0066]FIG. 7 illustrates a background area, a foreground area, a mixedarea, a covered background area, and an uncovered background area.

[0067]FIG. 8 illustrates a model obtained by expanding in the timedirection the pixel values of pixels aligned side-by-side in an imageobtained by image-capturing an object corresponding to a stationaryforeground and an the object corresponding to a stationary background.

[0068]FIG. 9 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0069]FIG. 10 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0070]FIG. 11 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0071]FIG. 12 illustrates an example in which pixels in a foregroundarea, a background area, and a mixed area are extracted.

[0072]FIG. 13 illustrates the relationships between pixels and a modelobtained by expanding the pixel values in the time direction.

[0073]FIG. 14 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0074]FIG. 15 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0075]FIG. 16 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0076]FIG. 17 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0077]FIG. 18 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0078]FIG. 19 is a flowchart illustrating the processing for adjustingthe amount of motion blur.

[0079]FIG. 20 is a block diagram illustrating an example of theconfiguration of the area specifying unit 103.

[0080]FIG. 21 illustrates an image when an object corresponding to aforeground is moving.

[0081]FIG. 22 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0082]FIG. 23 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0083]FIG. 24 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0084]FIG. 25 illustrates the conditions for determining the area.

[0085]FIG. 26A illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0086]FIG. 26B illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0087]FIG. 26C illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0088]FIG. 26D illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0089]FIG. 27 illustrates an example of the result obtained byspecifying the area by the area specifying unit 103.

[0090]FIG. 28 is a flowchart illustrating the area specifyingprocessing.

[0091]FIG. 29 is a block diagram illustrating an example of theconfiguration of the area specifying unit 103.

[0092]FIG. 30 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0093]FIG. 31 illustrates an example of a background image.

[0094]FIG. 32 is a block diagram illustrating the configuration of abinary-object-image extracting portion 302.

[0095]FIG. 33A illustrates the calculation of a correlation value.

[0096]FIG. 33B illustrates the calculation of a correlation value.

[0097]FIG. 34A illustrates the calculation of a correlation value.

[0098]FIG. 34B illustrates the calculation of a correlation value.

[0099]FIG. 35 illustrates an example of the binary object image.

[0100]FIG. 36 is a block diagram illustrating the configuration of atime change detector 303.

[0101]FIG. 37 illustrates determinations made by an area determiningportion 342.

[0102]FIG. 38 illustrates an example of determinations made by the timechange detector 303.

[0103]FIG. 39 is a flowchart illustrating the area specifying processingperformed by the area specifying unit 103.

[0104]FIG. 40 is a flowchart illustrating details of the area specifyingprocessing.

[0105]FIG. 41 is a block diagram illustrating still anotherconfiguration of the area specifying unit 103.

[0106]FIG. 42 is a block diagram illustrating the configuration of arobust-processing portion 361.

[0107]FIG. 43 illustrates motion compensation performed by a motioncompensator 381.

[0108]FIG. 44 illustrates motion compensation performed by the motioncompensator 381.

[0109]FIG. 45 is a flowchart illustrating the area specifyingprocessing.

[0110]FIG. 46 is a flowchart illustrating details of the robustprocessing.

[0111]FIG. 47 is a block diagram illustrating an example of theconfiguration of a mixture-ratio calculator 104.

[0112]FIG. 48 illustrates an example of the ideal mixture-ratio α.

[0113]FIG. 49 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0114]FIG. 50 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0115]FIG. 51 illustrates approximation using a correlation offoreground components.

[0116]FIG. 52 illustrates the relationship among C, N, and P.

[0117]FIG. 53 is a block diagram illustrating the configuration of themixture-ratio estimation processor 401.

[0118]FIG. 54 illustrates an example of an estimated mixture ratio.

[0119]FIG. 55 is a block diagram illustrating the configuration of themixture-ratio calculator 104.

[0120]FIG. 56 is a flowchart illustrating the mixture-ratio calculationprocessing.

[0121]FIG. 57 is a flowchart illustrating the processing for calculatingthe estimated mixture ratio.

[0122]FIG. 58 illustrates a straight line for approximating the mixtureratio α.

[0123]FIG. 59 illustrates a plane for approximating the mixture ratio α.

[0124]FIG. 60 illustrates the relationships of the pixels in a pluralityof frames when the mixture ratio α is calculated.

[0125]FIG. 61 is a block diagram illustrating another configuration ofthe estimated-mixture-ratio processor 401.

[0126]FIG. 62 illustrates an example of an estimated mixture ratio.

[0127]FIG. 63 is a flowchart illustrating the mixture-ratio estimatingprocessing by using a model corresponding to a covered background area.

[0128]FIG. 64 is a block diagram illustrating an example of theconfiguration of a foreground/background separator 105.

[0129]FIG. 65A illustrates an input image, a foreground component image,and a background component image.

[0130]FIG. 65B illustrates a model of an input image, a foregroundcomponent image, and a background component image.

[0131]FIG. 66 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0132]FIG. 67 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0133]FIG. 68 illustrates a model in which pixel values are expanded inthe time direction and the period corresponding to the shutter time isdivided.

[0134]FIG. 69 is a block diagram illustrating an example of theconfiguration of a separating portion 601.

[0135]FIG. 70A illustrates an example of a separated foregroundcomponent image.

[0136]FIG. 70B illustrates an example of a separated backgroundcomponent image.

[0137]FIG. 71 is a flowchart illustrating the processing for separatinga foreground and a background.

[0138]FIG. 72 is a block diagram illustrating an example of theconfiguration of a motion-blur adjusting unit 106.

[0139]FIG. 73 illustrates the unit of processing.

[0140]FIG. 74 illustrates a model in which the pixel values of aforeground component image are expanded in the time direction and theperiod corresponding to the shutter time is divided.

[0141]FIG. 75 illustrates a model in which the pixel values of aforeground component image are expanded in the time direction and theperiod corresponding to the shutter time is divided.

[0142]FIG. 76 illustrates a model in which the pixel values of aforeground component image are expanded in the time direction and theperiod corresponding to the shutter time is divided.

[0143]FIG. 77 illustrates a model in which the pixel values of aforeground component image are expanded in the time direction and theperiod corresponding to the shutter time is divided.

[0144]FIG. 78 illustrates an example of another configuration of themotion-blur adjusting unit 106.

[0145]FIG. 79 is a flowchart illustrating the processing for adjustingthe amount of motion blur contained in a foreground component imageperformed by the motion-blur adjusting unit 106.

[0146]FIG. 80 is a block diagram illustrating an example of anotherconfiguration of the motion-blur adjusting unit 106.

[0147]FIG. 81 illustrates an example of a model in which therelationships between pixel values and foreground components areindicated.

[0148]FIG. 82 illustrates the calculation of foreground components.

[0149]FIG. 83 illustrates the calculation of foreground components.

[0150]FIG. 84 is a flowchart illustrating the processing for eliminatingmotion blur contained in a foreground.

[0151]FIG. 85 is a block diagram illustrating another configuration ofthe function of the image processing apparatus.

[0152]FIG. 86 illustrates the configuration of a synthesizer 1001.

[0153]FIG. 87 is a block diagram illustrating still anotherconfiguration of the function of the image processing apparatus.

[0154]FIG. 88 is a block diagram illustrating the configuration of amixture-ratio calculator 1101.

[0155]FIG. 89 is a block diagram illustrating the configuration of aforeground/background separator 1102.

[0156]FIG. 90 is a block diagram illustrating still anotherconfiguration of the function of the image processing apparatus.

[0157]FIG. 91 illustrates the configuration of a synthesizer 1201.

[0158]FIG. 92 shows an embodiment of an image processing apparatus forgenerating area information on the basis of input images which are inputas component signals.

[0159]FIG. 93 illustrates the relationship among component 1, component2, and component 3.

[0160]FIG. 94 is a flowchart illustrating the processing for determiningthe area using component signals.

[0161]FIG. 95 shows another embodiment of an image processing apparatusfor generating area information on the basis of input images which areinput as component signals.

[0162]FIG. 96 is a flowchart illustrating another processing fordetermining the area using component signals.

[0163]FIG. 97 shows still another embodiment of an image processingapparatus for generating area information on the basis of input imageswhich are input as component signals.

[0164]FIG. 98 shows the relationship between a space correlation and atime correlation in a stationary area and in a moving area.

[0165]FIG. 99 illustrates an example of the calculation of a spacecorrelation.

[0166]FIG. 100 illustrates an example of the calculation of a timecorrelation.

[0167]FIG. 101 illustrates a time correlation and a space correlation ina stationary area.

[0168]FIG. 102 illustrates a time correlation and a space correlation ina moving area.

[0169]FIG. 103 shows an example of an input image.

[0170]FIG. 104 shows results of the determination of a moving area or astationary area.

[0171]FIG. 105 shows results of the determination of a moving area or astationary area by using a block of 15×15 pixels as units.

[0172]FIG. 106 shows results of the determination of a foreground area,a background area, a covered background area, and an uncoveredbackground area.

[0173]FIG. 107 is a flowchart illustrating the processing fordetermining the area using component signals.

[0174]FIG. 108 shows an embodiment of an image processing apparatus forcalculating a mixture ratio on the basis of an input image and areainformation, which are input as component signals.

[0175]FIG. 109 illustrates the relationship of the mixture ratios ineach component signal.

[0176]FIG. 110 is a flowchart illustrating the processing forcalculating a mixture ratio using component signals.

[0177]FIG. 111 shows another embodiment of an image processing apparatusfor calculating a mixture ratio on the basis of input images and areainformation, which are input as component signals.

[0178]FIG. 112 is a flowchart illustrating another processing fordetermining a mixture ratio using component signals.

[0179]FIG. 113 shows still another embodiment of an image processingapparatus for calculating a mixture ratio on the basis of input imagesand area information, which are input as component signals.

[0180]FIG. 114 is a block diagram illustrating the configuration of amixture-ratio calculator 1421.

[0181]FIG. 115 illustrates still another processing for calculating amixture ratio on the basis of input images and area information, whichare input as component signals.

BEST MODE FOR CARRYING OUT THE INVENTION

[0182]FIG. 1 shows an embodiment of an image processing apparatus of thepresent invention. A CPU (Central Processing Unit) 21 executes varioustypes of processing according to programs stored in a ROM (Read OnlyMemory) 22 or in a storage unit 28. Programs executed by the CPU 21 anddata are stored in a RAM (Random Access Memory) 23 as required. The CPU21, the ROM 22, and the RAM 23 are connected to each other by a bus 24.

[0183] An input/output interface 25 is also connected to the CPU 21 viathe bus 24. An input unit 26, which is formed of a keyboard, a mouse, amicrophone, and so on, and an output unit 27, which is formed of adisplay, a speaker, and so on, are connected to the input/outputinterface 25. The CPU 21 executes various types of processing inresponse to a command input from the input unit 26. The CPU 21 thenoutputs an image or sound obtained as a result of the processing to theoutput unit 27.

[0184] The storage unit 28 connected to the input/output interface 25 isformed of, for example, a hard disk, and stores programs executed by theCPU 21 and various types of data. A communication unit 29 communicateswith an external device via the Internet or another network. In thisexample, the communication unit 29 serves as an obtaining unit forobtaining an output of a sensor.

[0185] Alternatively, a program may be obtained via the communicationunit 29 and stored in the storage unit 28.

[0186] A drive 30 connected to the input/output interface 25 drives amagnetic disk 51, an optical disc 52, a magneto-optical disk 53, asemiconductor memory 54, or the like, when such a recording medium isattached to the drive 30, and obtains a program or data stored in thecorresponding medium. The obtained program or data is transferred to thestorage unit 28 and stored therein if necessary.

[0187]FIG. 2 is a block diagram illustrating the image processingapparatus.

[0188] It does not matter whether the individual functions of the imageprocessing apparatus are implemented by hardware or software. That is,the block diagrams of this specification may be hardware block diagramsor software functional block diagrams.

[0189] In this specification, an image to be captured corresponding toan object in the real world is referred to as an image object.

[0190] An input image supplied to the image processing apparatus issupplied to an object extracting unit 101, an area specifying unit 103,a mixture-ratio calculator 104, and a foreground/background separator105.

[0191] The object extracting unit 101 extracts a rough image objectcorresponding to a foreground object contained in the input image, andsupplies the extracted image object to a motion detector 102. The objectextracting unit 101 detects, for example, an outline of the foregroundimage object contained in the input image so as to extract a rough imageobject corresponding to the foreground object.

[0192] The object extracting unit 101 extracts a rough image objectcorresponding to a background object contained in the input image, andsupplies the extracted image object to the motion detector 102. Theobject extracting unit 101 extracts a rough image object correspondingto the background object from, for example, the difference between theinput image and the extracted image object corresponding to theforeground object.

[0193] Alternatively, for example, the object extracting unit 101 mayextract the rough image object corresponding to the foreground objectand the rough image object corresponding to the background object fromthe difference between the background image stored in a built-inbackground memory and the input image.

[0194] The motion detector 102 calculates a motion vector of the roughlyextracted image object corresponding to the foreground object accordingto a technique, such as block matching, gradient, phase correlation, orpel-recursive technique, and supplies the calculated motion vector andthe motion-vector positional information (which is information forspecifying the positions of the pixels corresponding to the motionvector) to the area specifying unit 103, and a motion-blur adjustingunit 106.

[0195] The motion vector output from the motion detector 102 containsinformation corresponding to the amount of movement v.

[0196] The motion detector 102 may output the motion vector of eachimage object, together with the pixel positional information forspecifying the pixels of the image object, to the motion-blur adjustingunit 106.

[0197] The amount of movement v is a value indicating a positionalchange in an image corresponding to a moving object in units of thepixel pitch. For example, if an object image corresponding to aforeground is moving such that it is displayed at a position four pixelsaway from a reference frame when it is positioned in the subsequentframe, the amount of movement v of the object image corresponding to theforeground is 4.

[0198] The object extracting unit 101 and the motion detector 102 areneeded when adjusting the amount of motion blur corresponding to amoving object.

[0199] The area specifying unit 103 determines to which of a foregroundarea, a background area, or a mixed area each pixel of the input imagebelongs, and supplies information indicating to which area each pixelbelongs (hereinafter referred to as “area information”) to themixture-ratio calculator 104, the foreground/background separator 105,and the motion-blur adjusting unit 106.

[0200] The mixture-ratio calculator 104 calculates the mixture ratiocorresponding to the pixels contained in a mixed area 63 (hereinafterreferred to as the “mixture-ratio α”) based on the input image, and thearea information supplied from the area specifying unit 103, andsupplies the mixture ratio α to the foreground/background separator 105.

[0201] The mixture ratio α is a value indicating the ratio of the imagecomponents corresponding to the background object (hereinafter also bereferred to as “background components”) to the pixel value as expressedby equation (3), which is shown below.

[0202] The foreground/background separator 105 separates the input imageinto a foreground component image formed of only the image componentscorresponding to the foreground object (hereinafter also be referred toas “foreground components”) and a background component image formed ofonly the background components based on the area information suppliedfrom the area specifying unit 103 and the mixture ratio α supplied fromthe mixture-ratio calculator 104, and supplies the foreground componentimage to the motion-blur adjusting unit 106 and a selector 107. Theseparated foreground component image may be set as the final output. Amore precise foreground and background can be obtained compared to aknown method in which only a foreground and a background are specifiedwithout considering the mixed area.

[0203] The motion-blur adjusting unit 106 determines the unit ofprocessing indicating at least one pixel contained in the foregroundcomponent image based on the amount of movement v obtained from themotion vector and based on the area information. The unit of processingis data that specifies a group of pixels to be subjected to themotion-blur adjustments.

[0204] Based on the amount by which the motion blur is to be adjusted,which is input into the image processing apparatus, the foregroundcomponent image supplied from the foreground/background separator 105,the motion vector and the positional information thereof supplied fromthe motion detector 102, and the unit of processing, the motion-bluradjusting unit 106 adjusts the amount of motion blur contained in theforeground component image by removing, decreasing, or increasing themotion blur contained in the foreground component image. The motion-bluradjusting unit 106 then outputs the foreground component image in whichamount of motion blur is adjusted to the selector 107. It is notessential that the motion vector and the positional information thereofbe used.

[0205] Motion blur is a distortion contained in an image correspondingto a moving object caused by the movement of an object to be captured inthe real world and the image-capturing characteristics of the sensor.

[0206] The selector 107 selects one of the foreground component imagesupplied from the foreground/background separator 105 and the foregroundcomponent image in which the amount of motion blur is adjusted suppliedfrom the motion-blur adjusting unit 106 based on, for example, aselection signal reflecting a user's selection, and outputs the selectedforeground component image.

[0207] An input image supplied to the image processing apparatus isdiscussed below with reference to FIGS. 3 through 18.

[0208]FIG. 3 illustrates image capturing performed by a sensor. Thesensor is formed of, for example, a CCD (Charge-Coupled Device) videocamera provided with a CCD area sensor, which is a solid-stateimage-capturing device. An object 111 corresponding to a foreground inthe real world moves, for example, horizontally from the left to theright, between an object 112 corresponding to a background and thesensor.

[0209] The sensor captures the image of the object 111 corresponding tothe foreground together with the image of the object 112 correspondingto the background. The sensor outputs the captured image in units offrames. For example, the sensor outputs an image having 30 frames persecond. The exposure time of the sensor can be {fraction (1/30)} second.The exposure time is a period from when the sensor starts convertinginput light into electrical charge until when the conversion from theinput light to the electrical charge is finished. The exposure time isalso referred to as a “shutter time”.

[0210]FIG. 4 illustrates the arrangement of pixels. In FIG. 4, A throughI indicate the individual pixels. The pixels are disposed on a plane ofa corresponding image. One detection device corresponding to each pixelis disposed on the sensor. When the sensor performs image capturing,each detection device outputs a pixel value of the corresponding pixelforming the image. For example, the position of the detection device inthe X direction corresponds to the horizontal direction on the image,while the position of the detection device in the Y directioncorresponds to the vertical direction on the image.

[0211] As shown in FIG. 5, the detection device, which is, for example,a CCD, converts input light into electrical charge during a periodcorresponding to a shutter time, and stores the converted electricalcharge. The amount of charge is almost proportional to the intensity ofthe input light and the period for which the light is input. Thedetection device sequentially adds the electrical charge converted fromthe input light to the stored electrical charge during the periodcorresponding to the shutter time. That is, the detection deviceintegrates the input light during the period corresponding to theshutter time and stores the electrical charge corresponding to theamount integrated light. It can be considered that the detection devicehas an integrating function with respect to time.

[0212] The electrical charge stored in the detection device is convertedinto a voltage value by a circuit (not shown), and the voltage value isfurther converted into a pixel value, such as digital data, and isoutput. Accordingly, each pixel value output from the sensor is a valueprojected on a linear space, which is a result integrating a certainthree-dimensional portion of the object corresponding to the foregroundor the background with respect to the shutter time.

[0213] The image processing apparatus extracts significant informationembedded in the output signal, for example, the mixture ratio α, by thestorage operation of the sensor. The image processing apparatus adjuststhe amount of distortion, for example, the amount of motion blur, causedby the mixture of the foreground image object itself. The imageprocessing apparatus also adjusts the amount of distortion caused by themixture of the foreground image object and the background image object.

[0214]FIG. 6A illustrates an image obtained by capturing a objectcorresponding to a moving foreground and a object corresponding to astationary background. FIG. 6B illustrates a model corresponding to animage obtained by capturing a object corresponding to a movingforeground and a object corresponding to a stationary background.

[0215]FIG. 6A illustrates an image obtained by capturing a objectcorresponding to a moving foreground and a object corresponding to astationary background. In the example shown in FIG. 6A, the objectcorresponding to the foreground is moving horizontally from the left tothe right with respect to the screen.

[0216]FIG. 6B illustrates a model obtained by expanding pixel valuescorresponding to one line of the image shown in FIG. 6A in the timedirection. The horizontal direction shown in FIG. 6B corresponds to thespatial direction X in FIG. 6A.

[0217] The values of the pixels in the background area are formed onlyfrom the background components, that is, the image componentscorresponding to the background object. The values of the pixels in theforeground area are formed only from the foreground components, that is,the image components corresponding to the foreground object.

[0218] The values of the pixels of the mixed area are formed from thebackground components and the foreground components. Since the values ofthe pixels in the mixed area are formed from the background componentsand the foreground components, it may be referred to as a “distortionarea”. The mixed area is further classified into a covered backgroundarea and an uncovered background area.

[0219] The covered background area is a mixed area at a positioncorresponding to the leading end in the direction in which theforeground object is moving, where the background components aregradually covered with the foreground over time.

[0220] In contrast, the uncovered background area is a mixed areacorresponding to the trailing end in the direction in which theforeground object is moving, where the background components graduallyappear over time.

[0221] As discussed above, the image containing the foreground area, thebackground area, or the covered background area or the uncoveredbackground area is input into the area specifying unit 103, themixture-ratio calculator 104, and the foreground/background separator105 as the input image.

[0222]FIG. 7 illustrates the background area, the foreground area, themixed area, the covered background area, and the uncovered backgroundarea discussed above. In the areas corresponding to the image shown inFIG. 6A, the background area is a stationary portion, the foregroundarea is a moving portion, the covered background area of the mixed areais a portion that changes from the background to the foreground, and theuncovered background area of the mixed area is a portion that changesfrom the foreground to the background.

[0223]FIG. 8 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels aligned side-by-side in theimage obtained by capturing the image of the object corresponding to thestationary foreground and the image of the object corresponding to thestationary background. For example, as the pixels aligned side-by-side,pixels arranged in one line on the screen can be selected.

[0224] The pixel values indicated by F01 through F04 shown in FIG. 8 arevalues of the pixels corresponding to the object of the stationaryforeground. The pixel values indicated by B01 through B04 shown in FIG.8 are values of the pixels corresponding to the object of the stationarybackground.

[0225] Time elapses from the top to the bottom in FIG. 8 in the verticaldirection in FIG. 8. The position at the top side of the rectangle inFIG. 8 corresponds to the time at which the sensor starts convertinginput light into electrical charge, and the position at the bottom sideof the rectangle in FIG. 8 corresponds to the time at which theconversion from the input light into the electrical charge is finished.That is, the distance from the top side to the bottom side of therectangle in FIG. 8 corresponds to the shutter time.

[0226] The pixels shown in FIG. 8 are described below assuming that, forexample, the shutter time is equal to the frame size.

[0227] The horizontal direction in FIG. 8 corresponds to the spatialdirection X in FIG. 6A. More specifically, in the example shown in FIG.8, the distance from the left side of the rectangle indicated by “F01”in FIG. 8 to the right side of the rectangle indicated by “B04” is eighttimes the pixel pitch, i.e., eight consecutive pixels.

[0228] When the foreground object and the background object arestationary, the light input into the sensor does not change during theperiod corresponding to the shutter time.

[0229] The period corresponding to the shutter time is divided into twoor more portions of equal periods. For example, if the number of virtualdivided portions is 4, the model shown in FIG. 8 can be represented bythe model shown in FIG. 9. The number of virtual divided portions can beset according to the amount of movement v of the object corresponding tothe foreground within the shutter time. For example, the number ofvirtual divided portions is set to 4 when the amount of movement v is 4,and the period corresponding to the shutter time is divided into fourportions.

[0230] The uppermost line in FIG. 9 corresponds to the first dividedperiod from when the shutter has opened. The second line in FIG. 9corresponds to the second divided period from when the shutter hasopened. The third line in FIG. 9 corresponds to the third divided periodfrom when the shutter has opened. The fourth line in FIG. 9 correspondsto the fourth divided period from when the shutter has opened.

[0231] The shutter time divided in accordance with the amount ofmovement v is also hereinafter referred to as the “shutter time/v”.

[0232] When the object corresponding to the foreground is stationary,the light input into the sensor does not change, and thus, theforeground component F01/v is equal to the value obtained by dividingthe pixel value F01 by the number of virtual divided portions.Similarly, when the object corresponding to the foreground isstationary, the foreground component F02/v is equal to the valueobtained by dividing the pixel value F02 by the number of virtualdivided portions, the foreground component F03/v is equal to the valueobtained by dividing the pixel value F03 by the number of virtualdivided portions, and the foreground component F04/v is equal to thevalue obtained by dividing the pixel value F04 by the number of virtualdivided portions.

[0233] When the object corresponding to the background is stationary,the light input into the sensor does not change, and thus, thebackground component B01/v is equal to the value obtained by dividingthe pixel value B01 by the number of virtual divided portions.Similarly, when the object corresponding to the background isstationary, the background component B02/v is equal to the valueobtained by dividing the pixel value B02 by the number of virtualdivided portions, the background component B03/v is equal to the valueobtained by dividing the pixel value B03 by the number of virtualdivided portions, and the background component B04/v is equal to thevalue obtained by dividing the pixel value B04 by the number of virtualdivided portions.

[0234] More specifically, when the object corresponding to theforeground is stationary, the light corresponding to the foregroundobject input into the sensor does not change during the periodcorresponding to the shutter time. Accordingly, the foreground componentF01/v corresponding to the first portion of the shutter time/v from whenthe shutter has opened, the foreground component F01/v corresponding tothe second portion of the shutter time/v from when the shutter hasopened, the foreground component F01/v corresponding to the thirdportion of the shutter time/v from when the shutter has opened, and theforeground component F01/v corresponding to the fourth portion of theshutter time/v from when the shutter has opened become the same value.The same applies to F02/v through F04/v, as in the case of F01/v.

[0235] When the object corresponding to the background is stationary,the light corresponding to the background object input into the sensordoes not change during the period corresponding to the shutter time.Accordingly, the background component B01/v corresponding to the firstportion of the shutter time/v from when the shutter has opened, thebackground component B01/v corresponding to the second portion of theshutter time/v from when the shutter has opened, the backgroundcomponent B01/v corresponding to the third portion of the shutter time/vfrom when the shutter has opened, and the background component B01/vcorresponding to the fourth portion of the shutter time/v from when theshutter has opened become the same value. The same applies to B02/vthrough B04/v.

[0236] A description is given of the case in which the objectcorresponding to the foreground is moving and the object correspondingto the background is stationary.

[0237]FIG. 10 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels in one line, including acovered background area, when the object corresponding to the foregroundis moving to the right in FIG. 10. In FIG. 10, the amount of movement vis 4. Since one frame is a short period, it can be assumed that theobject corresponding to the foreground is a rigid body moving withconstant velocity. In FIG. 10, the object image corresponding to theforeground is moving such that it is positioned four pixels to the rightwith respect to a reference frame when it is displayed in the subsequentframe.

[0238] In FIG. 10, the pixels from the leftmost pixel to the fourthpixel belong to the foreground area. In FIG. 10, the pixels from thefifth pixel to the seventh pixel from the left belong to the mixed area,which is the covered background area. In FIG. 10, the rightmost pixelbelongs to the background area.

[0239] The object corresponding to the foreground is moving such that itgradually covers the object corresponding to the background over time.Accordingly, the components contained in the pixel values of the pixelsbelonging to the covered background area change from the backgroundcomponents to the foreground components at a certain time during theperiod corresponding to the shutter time.

[0240] For example, the pixel value M surrounded by the thick frame inFIG. 10 is expressed by equation (1) below.

M=B02/v+B02/v+F07/v+F06/v  (1)

[0241] For example, the fifth pixel from the left contains a backgroundcomponent corresponding to one portion of the shutter time/v andforeground components corresponding to three portions of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 1/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains background components correspondingto three portions of the shutter time/v and a foreground componentcorresponding to one portion of the shutter time/v, and thus, themixture ratio α of the fifth pixel from the left is 3/4.

[0242] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF07/v of the fourth pixel from the left in FIG. 10 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the fifth pixel from the left inFIG. 10 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F07/vis equal to the foreground component of the sixth pixel from the left inFIG. 10 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the seventhpixel from the left in FIG. 10 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0243] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF06/v of the third pixel from the left in FIG. 10 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the fourth pixel from the left inFIG. 10 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F06/vis equal to the foreground component of the fifth pixel from the left inFIG. 10 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the sixthpixel from the left in FIG. 10 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0244] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF05/v of the second pixel from the left in FIG. 10 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the third pixel from the left inFIG. 10 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F05/vis equal to the foreground component of the fourth pixel from the leftin FIG. 10 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the fifthpixel from the left in FIG. 10 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0245] It can be assumed that the object corresponding to the foregroundis a rigid body, and the foreground object is moving with constantvelocity such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, for example, the foreground componentF04/v of the left most pixel in FIG. 10 corresponding to the firstportion of the shutter time/v from when the shutter has opened is equalto the foreground component of the second pixel from the left in FIG. 10corresponding to the second portion of the shutter time/v from when theshutter has opened. Similarly, the foreground component F04/v is equalto the foreground component of the third pixel from the left in FIG. 10corresponding to the third portion of the shutter time/v from when theshutter has opened, and the foreground component of the fourth pixelfrom the left in FIG. 10 corresponding to the fourth portion of theshutter time/v from when the shutter has opened.

[0246] Since the foreground area corresponding to the moving objectcontains motion blur as discussed above, it can also be referred to as a“distortion area”.

[0247]FIG. 11 illustrates a model obtained by expanding in the timedirection the pixel values of the pixels in one line including anuncovered background area when the object corresponding to theforeground is moving to the right in FIG. 11. In FIG. 11, the amount ofmovement v is 4. Since one frame is a short period, it can be assumedthat the object corresponding to the foreground is a rigid body movingwith constant velocity. In FIG. 11, the object image corresponding tothe foreground is moving to the right such that it is positioned fourpixels to the right with respect to a reference frame when it isdisplayed in the subsequent frame.

[0248] In FIG. 11, the pixels from the leftmost pixel to the fourthpixel belong to the background area. In FIG. 11, the pixels from thefifth pixel to the seventh pixels from the left belong to the mixedarea, which is an uncovered background area. In FIG. 11, the rightmostpixel belongs to the foreground area.

[0249] The object corresponding to the foreground which covers theobject corresponding to the background is moving such that it isgradually removed from the object corresponding to the background overtime. Accordingly, the components contained in the pixel values of thepixels belonging to the uncovered background area change from theforeground components to the background components at a certain time ofthe period corresponding to the shutter time.

[0250] For example, the pixel value M′ surrounded by the thick frame inFIG. 11 is expressed by equation (2).

M′=F02/v+F01/v+B26/v+B26/v  (2)

[0251] For example, the fifth pixel from the left contains backgroundcomponents corresponding to three portions of the shutter time/v and aforeground component corresponding to one shutter portion of the shuttertime/v, and thus, the mixture ratio α of the fifth pixel from the leftis 3/4. The sixth pixel from the left contains background componentscorresponding to two portions of the shutter time/v and foregroundcomponents corresponding to two portions of the shutter time/v, andthus, the mixture ratio α of the sixth pixel from the left is 1/2. Theseventh pixel from the left contains a background componentcorresponding to one portion of the shutter time/v and foregroundcomponents corresponding to three portions of the shutter time/v, andthus, the mixture ratio α of the seventh pixel from the left is 1/4.

[0252] When equations (1) and (2) are generalized, the pixel value M canbe expressed by equation (3): $\begin{matrix}{M = {{\alpha \cdot B} + {\sum\limits_{i}{F\quad {i/v}}}}} & (3)\end{matrix}$

[0253] where α is the mixture ratio, B indicates a pixel value of thebackground, and Fi/v designates a foreground component.

[0254] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement is 4. Accordingly, for example, the foreground componentF01/v of the fifth pixel from the left in FIG. 11 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the sixth pixel from the left inFIG. 11 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F01/vis equal to the foreground component of the seventh pixel from the leftin FIG. 11 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened, and the foreground component of the eighthpixel from the left in FIG. 11 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened.

[0255] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement v is 4. Accordingly, for example, the foreground componentF02/v of the sixth pixel from the left in FIG. 11 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the seventh pixel from the left inFIG. 11 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened. Similarly, the foreground component F02/vis equal to the foreground component of the eighth pixel from the leftin FIG. 11 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened.

[0256] It can be assumed that the object corresponding to the foregroundis a rigid body, which is moving with constant velocity, and the amountof movement v is 4. Accordingly, for example, the foreground componentF03/v of the seventh pixel from the left in FIG. 11 corresponding to thefirst portion of the shutter time/v from when the shutter has opened isequal to the foreground component of the eighth pixel from the left inFIG. 11 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened.

[0257] It has been described with reference to FIGS. 9 through 11 thatthe number of virtual divided portions is 4. The number of virtualdivided portions corresponds to the amount of movement v. Generally, theamount of movement v corresponds to the moving speed of the objectcorresponding to the foreground. For example, if the objectcorresponding to the foreground is moving such that it is displayed fourpixels to the right with respect to a certain frame when it ispositioned in the subsequent frame, the amount of movement v is set to4. The number of virtual divided portions is set to 4 in accordance withthe amount of movement v. Similarly, when the object corresponding tothe foreground is moving such that it is displayed six pixels to theleft with respect to a certain frame when it is positioned in thesubsequent frame, the amount of movement v is set to 6, and the numberof virtual divided portions is set to 6.

[0258]FIGS. 12 and 13 illustrate the relationship of the foregroundarea, the background area, and the mixed area which consists of acovered background or an uncovered background, which are discussedabove, to the foreground components and the background componentscorresponding to the divided periods of the shutter time.

[0259]FIG. 12 illustrates an example in which pixels in the foregroundarea, the background area, and the mixed area are extracted from animage containing a foreground corresponding to an object moving in frontof a stationary background. In the example shown in FIG. 12, the objectindicated by “A” corresponding to the foreground is horizontally movingwith respect to the screen.

[0260] Frame #n+1 is a frame subsequent to frame #n, and frame #n+2 is aframe subsequent to frame #n+1.

[0261] Pixels in the foreground area, the background area, and the mixedarea are extracted from one of frames #n through #n+2, and the amount ofmovement v is set to 4. A model obtained by expanding the pixel valuesof the extracted pixels in the time direction is shown in FIG. 13.

[0262] Since the object corresponding to the foreground is moving, thepixel values in the foreground area are formed of four differentforeground components corresponding to the shutter time/v. For example,the leftmost pixel of the pixels in the foreground area shown in FIG. 13consists of F01/v, F02/v, F03/v, and F04/v. That is, the pixels in theforeground contain motion blur.

[0263] Since the object corresponding to the background is stationary,light input into the sensor corresponding to the background during theshutter time does not change. In this case, the pixel values in thebackground area do not contain motion blur.

[0264] The pixel values in the mixed area consisting of a coveredbackground area or an uncovered background area are formed of foregroundcomponents and background components.

[0265] A description is given below of a model obtained by expanding inthe time direction the pixel values of the pixels which are alignedside-by-side in a plurality of frames and which are located at the samepositions when the frames are overlapped when the image corresponding tothe object is moving. For example, when the image corresponding to theobject is moving horizontally with respect to the screen, pixels alignedon the screen can be selected as the pixels aligned side-by-side.

[0266]FIG. 14 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a stationarybackground and which are located at the same positions when the framesare overlapped. Frame #n is the frame subsequent to frame #n−1, andframe #n+1 is the frame subsequent to frame #n. The same applies to theother frames.

[0267] The pixel values B01 through B12 shown in FIG. 14 are pixelvalues corresponding to the stationary background object. Since theobject corresponding to the background is stationary, the pixel valuesof the corresponding pixels in frame #n−1 through frame #n+1 do notchange. For example, the pixel in frame #n and the pixel in frame #n+1located at the corresponding position of the pixel having the pixelvalue B05 in frame #n−1 have the pixel value B05.

[0268]FIG. 15 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a foregroundthat is moving to the right in FIG. 15 together with an objectcorresponding to a stationary background and which are located at thesame positions when the frames are overlapped. The model shown in FIG.15 contains a covered background area.

[0269] In FIG. 15, it can be assumed that the object corresponding tothe foreground is a rigid body moving with constant velocity, and thatit is moving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4, and thenumber of virtual divided portions is 4.

[0270] For example, the foreground component of the leftmost pixel inframe #n−1 in FIG. 15 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F12/v, and the foregroundcomponent of the second pixel from the left in FIG. 15 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F12/v. The foreground component of the third pixel fromthe left in FIG. 15 corresponding to the third portion of the shuttertime/v from when the shutter has opened and the foreground component ofthe fourth pixel from the left in FIG. 15 corresponding to the fourthportion of the shutter time/v from when the shutter has opened areF12/v.

[0271] The foreground component of the leftmost pixel in frame #n−1 inFIG. 15 corresponding to the second portion of the shutter time/v fromwhen the shutter has opened is F11/v. The foreground component of thesecond pixel from the left in FIG. 15 corresponding to the third portionof the shutter time/v from when the shutter has opened is also F11/v.The foreground component of the third pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened is F11/v.

[0272] The foreground component of the leftmost pixel in frame #n−1 inFIG. 15 corresponding to the third portion of the shutter time/v fromwhen the shutter has opened is F10/v. The foreground component of thesecond pixel from the left in FIG. 15 corresponding to the fourthportion of the shutter time/v from when the shutter has opened is alsoF10/v. The foreground component of the leftmost pixel in frame #n−1 inFIG. 15 corresponding to the fourth portion of the shutter time/v fromwhen the shutter has opened is F09/v.

[0273] In frame #n−1 in FIG. 15, the leftmost pixel from the leftbelongs to the foreground area, and the second through fourth pixelsfrom the left belong to the mixed area, which is a covered backgroundarea.

[0274] The fifth through twelfth pixels from the left of frame #n−1 inFIG. 15 belong to the background area, and the pixel values thereof areB04 through B11, respectively.

[0275] The first through fifth pixels from the left in frame #n in FIG.15 belong to the foreground area. The foreground component in theshutter time/v in the foreground area of frame #n is any one of F05/vthrough F12/v.

[0276] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that the foreground image is displayed four pixels to the right inthe subsequent frame. Accordingly, the foreground component of the fifthpixel from the left of frame #n in FIG. 15 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the sixth pixel from the left in FIG. 15corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theseventh pixel from the left in FIG. 15 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the eighth pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

[0277] The foreground component of the fifth pixel from the left offrame #n in FIG. 15 corresponding to the second portion of the shuttertime/v from when the shutter has opened is F11/v. The foregroundcomponent of the sixth pixel from the left in FIG. 15 corresponding tothe third portion of the shutter time/v from when the shutter has openedis also F11/v. The foreground component of the seventh pixel from theleft in FIG. 15 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened is F11/v.

[0278] The foreground component of the fifth pixel from the left offrame #n in FIG. 15 corresponding to the third portion of the shuttertime/v from when the shutter has opened is F10/v. The foregroundcomponent of the sixth pixel from the left in FIG. 15 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is also F10/v. The foreground component of the fifth pixel fromthe left of frame #n in FIG. 15 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened is F09/v.

[0279] Since the object corresponding to the background is stationary,the background component of the sixth pixel from the left of frame #n inFIG. 15 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B05/v. The background components of theseventh pixel from the left of frame #n in FIG. 15 corresponding to thefirst and second portions of the shutter time/v from when the shutterhas opened are B06/v. The background components of the eighth pixel fromthe left of frame #n in FIG. 15 corresponding to the first through thirdportion of the shutter time/v from when the shutter has opened areB07/v.

[0280] In frame #n in FIG. 15, the sixth through eighth pixels from theleft belong to the mixed area, which is a covered background area.

[0281] The ninth through twelfth pixels from the left of frame #n inFIG. 15 belong to the background area, and the pixel values thereof areB08 through B11, respectively.

[0282] The first through ninth pixels from the left in frame #n+1 inFIG. 15 belong to the foreground area. The foreground component in theshutter time/v in the foreground area of frame #n+1 is any one of F01/vthrough F12/v.

[0283] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that the foreground image is displayed four pixels to the right inthe subsequent frame. Accordingly, the foreground component of the ninthpixel from the left of frame #n+1 in FIG. 15 corresponding to the firstportion of the shutter time/v from when the shutter has opened is F12/v,and the foreground component of the tenth pixel from the left in FIG. 15corresponding to the second portion of the shutter time/v from when theshutter has opened is also F12/v. The foreground component of theeleventh pixel from the left in FIG. 15 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 15corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F12/v.

[0284] The foreground component of the ninth pixel from the left offrame #n+1 in FIG. 15 corresponding to the second portion of the shuttertime/v from when the shutter has opened is F11/v. The foregroundcomponent of the tenth pixel from the left in FIG. 15 corresponding tothe third portion of the shutter time/v from when the shutter has openedis also F11/v. The foreground component of the eleventh pixel from theleft in FIG. 15 corresponding to the fourth portion of the shuttertime/v from when the shutter has opened is F11/v.

[0285] The foreground component of the ninth pixel from the left offrame #n+1 in FIG. 15 corresponding to the third portion of the shuttertime/v from when the shutter has opened is F10/v. The foregroundcomponent of the tenth pixel from the left in FIG. 15 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is also F10/v. The foreground component of the ninth pixel fromthe left of frame #n+1 in FIG. 15 corresponding to the fourth portion ofthe shutter time/v from when the shutter has opened is F09/v.

[0286] Since the object corresponding to the background is stationary,the background component of the tenth pixel from the left of frame #n+1in FIG. 15 corresponding to the first portion of the shutter time/v fromwhen the shutter has opened is B09/v. The background components of theeleventh pixel from the left of frame #n+1 in FIG. 15 corresponding tothe first and second portions of the shutter time/v from when theshutter has opened are B10/v. The background components of the twelfthpixel from the left of frame #n+1 in FIG. 15 corresponding to the firstthrough third portion of the shutter time/v from when the shutter hasopened are B11/v.

[0287] In frame #n+1 in FIG. 15, the tenth through twelfth pixels fromthe left belong to the mixed area, which is a covered background area.

[0288]FIG. 16 illustrates a model of an image obtained by extracting theforeground components from the pixel values shown in FIG. 15.

[0289]FIG. 17 illustrates a model obtained by expanding in the timedirection the pixels which are aligned side-by-side in three frames ofan image obtained by capturing an object corresponding to a foregroundthat is moving to the right in FIG. 17 together with an objectcorresponding to a stationary background and which are located at thesame positions when the frames are overlapped. The model shown in FIG.17 contains an uncovered background area.

[0290] In FIG. 17, it can be assumed that the object corresponding tothe foreground is a rigid body moving with constant velocity, and thatit is moving such that it is displayed four pixels to the right in thesubsequent frame. Accordingly, the amount of movement v is 4.

[0291] For example, the foreground component of the leftmost pixel inframe #n−1 in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F13/v, and the foregroundcomponent of the second pixel from the left in FIG. 17 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F13/v. The foreground component of the third pixel fromthe left in FIG. 19 corresponding to the third portion of the shuttertime/v from when the shutter has opened and the foreground component ofthe fourth pixel from the left in FIG. 17 corresponding to the fourthportion of the shutter time/v from when the shutter has opened areF13/v.

[0292] The foreground component of the second pixel from the left offrame #n−1 in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the third pixel from the left in FIG. 17 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the third pixel fromthe left in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0293] Since the object corresponding to the background is stationary,the background components of the leftmost pixel in frame #n−1 in FIG. 17corresponding to the second through fourth portions of the shuttertime/v from when the shutter has opened are B25/v. The backgroundcomponents of the second pixel from the left of frame #n−1 in FIG. 17corresponding to the third and fourth portions of the shutter time/vfrom when the shutter has opened are B26/v. The background component ofthe third pixel from the left of frame #n−1 in FIG. 17 corresponding tothe fourth portion of the shutter time/v from when the shutter hasopened is B27/v.

[0294] In frame #n−1 in FIG. 17, the leftmost pixel through the thirdpixel belong to the mixed area, which is an uncovered background area.

[0295] The fourth through twelfth pixels from the left of frame #n−1 inFIG. 17 belong to the foreground area. The foreground component of theframe is any one of F13/v through F24/v.

[0296] The leftmost pixel through the fourth pixel from the left offrame #n in FIG. 17 belong to the background area, and the pixel valuesthereof are B25 through B28, respectively.

[0297] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, the foreground component of the fifth pixel from theleft of frame #n in FIG. 17 corresponding to the first portion of theshutter time/v from when the shutter has opened is F13/v, and theforeground component of the sixth pixel from the left in FIG. 17corresponding to the second portion of the shutter time/v from when theshutter has opened is also F13/v. The foreground component of theseventh pixel from the left in FIG. 17 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the eighth pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F13/v.

[0298] The foreground component of the sixth pixel from the left offrame #n in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the seventh pixel from the left in FIG. 17 corresponding tothe second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the eighth pixel fromthe left in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0299] Since the object corresponding to the background is stationary,the background components of the fifth pixel from the left of frame #nin FIG. 17 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B29/v. Thebackground components of the sixth pixel from the left of frame #n inFIG. 17 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B30/v. The backgroundcomponent of the seventh pixel from the left of frame #n in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B31/v.

[0300] In frame #n in FIG. 17, the fifth pixel through the seventh pixelfrom the left belong to the mixed area, which is an uncovered backgroundarea.

[0301] The eighth through twelfth pixels from the left of frame #n inFIG. 17 belong to the foreground area. The value in the foreground areaof frame #n corresponding to the period of the shutter time/v is any oneof F13/v through F20/v.

[0302] The leftmost pixel through the eighth pixel from the left offrame #n+1 in FIG. 17 belong to the background area, and the pixelvalues thereof are B25 through B32, respectively.

[0303] It can be assumed that the object corresponding to the foregroundis a rigid body moving with constant velocity, and that it is movingsuch that it is displayed four pixels to the right in the subsequentframe. Accordingly, the foreground component of the ninth pixel from theleft of frame #n+1 in FIG. 17 corresponding to the first portion of theshutter time/v from when the shutter has opened is F13/v, and theforeground component of the tenth pixel from the left in FIG. 17corresponding to the second portion of the shutter time/v from when theshutter has opened is also F13/v. The foreground component of theeleventh pixel from the left in FIG. 17 corresponding to the thirdportion of the shutter time/v from when the shutter has opened and theforeground component of the twelfth pixel from the left in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened are F13/v.

[0304] The foreground component of the tenth pixel from the left offrame #n+1 in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F14/v. The foregroundcomponent of the eleventh pixel from the left in FIG. 17 correspondingto the second portion of the shutter time/v from when the shutter hasopened is also F14/v. The foreground component of the twelfth pixel fromthe left in FIG. 17 corresponding to the first portion of the shuttertime/v from when the shutter has opened is F15/v.

[0305] Since the object corresponding to the background is stationary,the background components of the ninth pixel from the left of frame #n+1in FIG. 17 corresponding to the second through fourth portions of theshutter time/v from when the shutter has opened are B33/v. Thebackground components of the tenth pixel from the left of frame #n+1 inFIG. 17 corresponding to the third and fourth portions of the shuttertime/v from when the shutter has opened are B34/v. The backgroundcomponent of the eleventh pixel from the left of frame #n+1 in FIG. 17corresponding to the fourth portion of the shutter time/v from when theshutter has opened is B35/v.

[0306] In frame #n+1 in FIG. 17, the ninth through eleventh pixels fromthe left in FIG. 17 belong to the mixed area, which is an uncoveredbackground area.

[0307] The twelfth pixel from the left of frame #n+1 in FIG. 17 belongsto the foreground area. The foreground component in the shutter time/vin the foreground area of frame #n+1 is any one of F13/v through F16/v,respectively.

[0308]FIG. 18 illustrates a model of an image obtained by extracting theforeground components from the pixel values shown in FIG. 17.

[0309] Referring back to FIG. 2, the area specifying unit 103 specifiesflags indicating to which of a foreground area, a background area, acovered background area, or an uncovered background area the individualpixels of the input image belong by using the pixel values of aplurality of frames, and supplies the flags to the mixture ratiocalculator 104 and the motion-blur adjusting unit 106 as the areainformation.

[0310] The mixture-ratio calculator 104 calculates the mixture ratio αfor each pixel contained in the mixed area based on the pixel values ofa plurality of frames and the area information, and supplies thecalculated mixture-ratio α to the foreground/background separator 105.

[0311] The foreground/background separator 105 extracts the foregroundcomponent image consisting of only the foreground components based onthe pixel values of a plurality of frames, the area information, and themixture ratio α, and supplies the foreground component image to themotion-blur adjusting unit 106.

[0312] The motion-blur adjusting unit 106 adjusts the amount of motionblur contained in the foreground component image based on the foregroundcomponent image supplied from the foreground/background separator 105,the motion vector supplied from the motion detector 102, and the areainformation supplied from the area specifying unit 103, and then outputsthe foreground component image in which motion blur is adjusted.

[0313] The processing for adjusting the amount of motion blur performedby the image processing apparatus is described below with reference tothe flowchart of FIG. 19. In step S11, the area specifying unit 103executes area specifying processing, based on an input image, forgenerating area information indicating to which of a foreground area, abackground area, a covered background area, or an uncovered backgroundarea each pixel of the input image belongs. Details of the areaspecifying processing are given below. The area specifying unit 103supplies the generated area information to the mixture-ratio calculator104.

[0314] In step S11, the area specifying unit 103 may generate, based onthe input image, area information indicating to which of the foregroundarea, the background area, or the mixed area (regardless of whether eachpixel belongs to a covered background area or an uncovered backgroundarea) each pixel of the input image belongs. In this case, theforeground/background separator 105 and the motion-blur adjusting unit106 determine based on the direction of the motion vector whether themixed area is a covered background area or an uncovered background area.For example, if the input image is disposed in the order of theforeground area, the mixed area, and the background area in thedirection of the motion vector, it is determined that the mixed area isa covered background area. If the input image is disposed in the orderof the background area, the mixed area, and the foreground area in thedirection of the motion vector, it is determined that the mixed area isan uncovered background area.

[0315] In step S12, the mixture-ratio calculator 104 calculates themixture ratio α for each pixel contained in the mixed area based on theinput image, and the area information. Details of the mixture-ratiocalculating processing are given below. The mixture-ratio calculator 104supplies the calculated mixture-ratio α to the foreground/backgroundseparator 105.

[0316] In step S13, the foreground/background separator 105 extracts theforeground components from the input image based on the area informationand the mixture ratio α, and supplies the foreground components to themotion-blur adjusting unit 106 as the foreground component image.

[0317] In step S14, the motion-blur adjusting unit 106 generates, basedon the motion vector and the area information, the unit of processingthat indicates the positions of consecutive pixels disposed in themoving direction and belonging to any of the uncovered background area,the foreground area, and the covered background area, and adjusts theamount of motion blur contained in the foreground componentscorresponding to the unit of processing. Details of the processing foradjusting the amount of motion blur are given below.

[0318] In step S15, the image processing apparatus determines whetherthe processing is finished for the whole screen. If it is determinedthat the processing is not finished for the whole screen, the processproceeds to step S14, and the processing for adjusting the amount ofmotion blur for the foreground components corresponding to the unit ofprocessing is repeated.

[0319] If it is determined in step S15 that the processing is finishedfor the whole screen, the processing is completed.

[0320] In this manner, the image processing apparatus is capable ofadjusting the amount of motion blur contained in the foreground byseparating the foreground and the background. That is, the imageprocessing apparatus is capable of adjusting the amount of motion blurcontained in sampled data indicating the pixel values of the foregroundpixels.

[0321] The configuration of each of the area specifying unit 103, themixture-ratio calculator 104, the foreground/background separator 105,and the motion-blur adjusting unit 106 is described below.

[0322]FIG. 20 is a block diagram illustrating an example of theconfiguration of the area specifying unit 103. The area specifying unit103 configured as shown in FIG. 20 does not use a motion vector. A framememory 201 stores an input image in units of frames. When the image tobe processed is frame #n, the frame memory 201 stores frame #n−2, whichis the frame two frames before frame #n, frame #n−1 which is the frameone frame before frame #n, frame #n, frame #n+1, which is the frame oneframe after frame #n, frame #n+2, which is the frame two frames afterframe #n.

[0323] A stationary/moving determining portion 202-1 reads the pixelvalue of the pixel in frame #n+2 located at the same position as adesignated pixel in frame #n in which the area to which the pixelbelongs is determined, and reads the pixel value of the pixel in frame#n+1 located at the same position of the designated pixel in frame #nfrom the frame memory 201, and calculates the absolute value of thedifference between the read pixel values. The stationary/movingdetermining portion 202-1 determines whether the absolute value of thedifference between the pixel value of frame #n+2 and the pixel value offrame #n+1 is greater than a preset threshold Th. If it is determinedthat the difference is greater than the threshold Th, astationary/moving determination indicating “moving” is supplied to anarea determining portion 203-1. If it is determined that the absolutevalue of the difference between the pixel value of the pixel in frame#n+2 and the pixel value of the pixel in frame #n+1 is smaller than orequal to the threshold Th, the stationary/moving determining portion202-1 supplies a stationary/moving determination indicating “stationary”to the area determining portion 203-1.

[0324] A stationary/moving determining portion 202-2 reads the pixelvalue of a designated pixel in frame #n in which the area to which thepixel belongs is determined, and reads the pixel value of the pixel inframe #n+1 located at the same position as the designated pixel in frame#n from the frame memory 201, and calculates the absolute value of thedifference between the pixel values. The stationary/moving determiningportion 202-2 determines whether the absolute value of the differencebetween the pixel value of frame #n+1 and the pixel value of frame #n isgreater than a preset threshold Th. If it is determined that theabsolute value of the difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-1 and an areadetermining portion 203-2. If it is determined that the absolute valueof the difference between the pixel value of the pixel in frame #n+1 andthe pixel value of the pixel in frame #n is smaller than or equal to thethreshold Th, the stationary/moving determining portion 202-2 supplies astationary/moving determination indicating “stationary” to the areadetermining portion 203-1 and the area determining portion 203-2.

[0325] A stationary/moving determining portion 202-3 reads the pixelvalue of a designated pixel in frame #n in which the area to which thepixel belongs is determined, and reads the pixel value of the pixel inframe #n−1 located at the same position as the designated pixel in frame#n from the frame memory 201, and calculates the absolute value of thedifference between the pixel values. The stationary/moving determiningportion 202-3 determines whether the absolute value of the differencebetween the pixel value of frame #n and the pixel value of frame #n−1 isgreater than a preset threshold Th. If it is determined that theabsolute value of the difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-2 and an areadetermining portion 203-3. If it is determined that the absolute valueof the difference between the pixel value of the pixel in frame #n andthe pixel value of the pixel in frame #n−1 is smaller than or equal tothe threshold Th, the stationary/moving determining portion 202-3supplies a stationary/moving determination indicating “stationary” tothe area determining portion 203-2 and the area determining portion203-3.

[0326] A stationary/moving determining portion 202-4 reads the pixelvalue of the pixel in frame #n−1 located at the same position as adesignated pixel in frame #n in which the area to which the pixelbelongs is determined, and reads the pixel value of the pixel in frame#n−2 located at the same position as the designated pixel in frame #nfrom the frame memory 201, and calculates the absolute value of thedifference between the pixel values. The stationary/moving determiningportion 202-4 determines whether the absolute value of the differencebetween the pixel value of frame #n−1 and the pixel value of frame #n−2is greater than a preset threshold Th. If it is determined that theabsolute value of the difference between the pixel values is greaterthan the threshold Th, a stationary/moving determination indicating“moving” is supplied to the area determining portion 203-3. If it isdetermined that the absolute value of the difference between the pixelvalue of the pixel in frame #n−1 and the pixel value of the pixel inframe #n−2 is smaller than or equal to the threshold Th, thestationary/moving determining portion 202-4 supplies a stationary/movingdetermination indicating “stationary” to the area determining portion203-3.

[0327] When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving”, the areadetermining portion 203-1 determines that the designated pixel in frame#n belongs to an uncovered background area, and sets “1”, whichindicates that the designated pixel belongs to an uncovered backgroundarea, in an uncovered-background-area determining flag associated withthe designated pixel.

[0328] When the stationary/moving determination supplied from thestationary/moving determining portion 202-1 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-2 indicates “stationary”, the area specifyingunit 203-1 determines that the designated pixel in frame #n does notbelong to an uncovered background area, and sets “0”, which indicatesthat the designated pixel does not belong to an uncovered backgroundarea, in the uncovered-background-area determining flag associated withthe designated pixel.

[0329] The area determining portion 203-1 supplies theuncovered-background-area determining flag in which “1” or “0” is set asdiscussed above to a determining-flag-storing frame memory 204.

[0330] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “stationary” andwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicate “stationary”, thearea determining portion 203-2 determines that the designated pixel inframe #n belongs to the stationary area, and sets “1”, which indicatesthat the pixel belongs to the stationary area, in a stationary-areadetermining flag associated with the designated pixel.

[0331] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving” or whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicate “moving”, the area determiningportion 203-2 determines that the designated pixel in frame #n does notbelong to the stationary area, and sets “0”, which indicates that thepixel does not belong to the stationary area, in the stationary-areadetermining flag associated with the designated pixel.

[0332] The area determining portion 203-2 supplies the stationary-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 204.

[0333] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-3 indicate “moving”, the area determiningportion 203-2 determines that the designated pixel in frame #n belongsto the moving area, and sets “1”, which indicates that the designatedpixel belongs to the moving area, in a moving-area determining flagassociated with the designated pixel.

[0334] When the stationary/moving determination supplied from thestationary/moving determining portion 202-2 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicate “stationary”, thearea determining portion 203-2 determines that the designated pixel inframe #n does not belong to the moving area, and sets “0”, whichindicates that the pixel does not belong to the moving area, in themoving-area determining flag associated with the designated pixel.

[0335] The area determining portion 203-2 supplies the moving-areadetermining flag in which “1” or “0” is set as discussed above to thedetermining-flag-storing frame memory 204.

[0336] When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates “moving” and whenthe stationary/moving determination supplied from the stationary/movingdetermining portion 202-4 indicate “stationary”, the area determiningportion 203-3 determines that the designated pixel in frame #n belongsto a covered background area, and sets “1”, which indicates that thedesignated pixel belongs to the covered background area, in acovered-background-area determining flag associated with the designatedpixel.

[0337] When the stationary/moving determination supplied from thestationary/moving determining portion 202-3 indicates “stationary” orwhen the stationary/moving determination supplied from thestationary/moving determining portion 202-4 indicate “moving”, the areadetermining portion 203-3 determines that the designated pixel in frame#n does not belong to a covered background area, and sets “0”, whichindicates that the designated pixel does not belong to a coveredbackground area, in the covered-background-area determining flagassociated with the designated pixel.

[0338] The area determining portion 203-3 supplies thecovered-background-area determining flag in which “1” or “0” is set asdiscussed above to the determining-flag-storing frame memory 204.

[0339] The determining-flag-storing frame memory 204 thus stores theuncovered-background-area determining flag supplied from the areadetermining portion 203-1, the stationary-area determining flag suppliedfrom the area determining portion 203-2, the moving-area determiningflag supplied from the area determining portion 203-2, and thecovered-background-area determining flag supplied from the areadetermining portion 203-3.

[0340] The determining-flag-storing frame memory 204 supplies theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag stored therein to a synthesizer205. The synthesizer 205 generates area information indicating to whichof the uncovered background area, the stationary area, the moving area,or the covered background area each pixel belongs based on theuncovered-background-area determining flag, the stationary-areadetermining flag, the moving-area determining flag, and thecovered-background-area determining flag supplied from thedetermining-flag-storing frame memory 204, and supplies the areainformation to a determining-flag-storing frame memory 206.

[0341] The determining-flag-storing frame memory 206 stores the areainformation supplied from the synthesizer 205, and also outputs the areainformation stored therein.

[0342] An example of the processing performed by the area specifyingunit 103 is described below with reference to FIGS. 21 through 25.

[0343] When the object corresponding to the foreground is moving, theposition of the image corresponding to the object on the screen changesin every frame. As shown in FIG. 21, the image corresponding to theobject located at the position indicated by Yn(x, y) in frame #n ispositioned at Yn+1(x, y) in frame #n+1, which is subsequent to frame #n.

[0344] A model obtained by expanding in the time direction the pixelvalues of the pixels aligned side-by-side in the moving direction of theimage corresponding to the foreground object is shown in FIG. 22. Forexample, if the moving direction of the image corresponding to theforeground object is horizontal with respect to the screen, the modelshown in FIG. 22 is a model obtained by expanding in the time directionthe pixel values of the pixels disposed on a line side-by-side.

[0345] In FIG. 22, the line in frame #n is equal to the line in frame#n+1.

[0346] The foreground components corresponding to the object containedin the second pixel to the thirteenth pixel from the left in frame #nare contained in the sixth pixel through the seventeenth pixel from theleft in frame #n+1.

[0347] In frame #n, the pixels belonging to the covered background areaare the eleventh through thirteenth pixels from the left, and the pixelsbelonging to the uncovered background area are the second through fourthpixels from the left. In frame #n+1, the pixels belonging to the coveredbackground area are the fifteenth through seventeenth pixels from theleft, and the pixels belonging to the uncovered background area are thesixth through eighth pixels from the left.

[0348] In the example shown in FIG. 22, since the foreground componentscontained in frame #n are moved by four pixels in frame #n+1, the amountof movement v is 4. The number of virtual divided portions is 4 inaccordance with the amount of movement v.

[0349] A description is now given of a change in pixel values of thepixels belonging to the mixed area in the frames before and after adesignated frame.

[0350] In FIG. 23, the pixels belonging to a covered background area inframe #n in which the background is stationary and the amount ofmovement v in the foreground is 4 are the fifteenth through seventeenthpixels from the left. Since the amount of movement v is 4, the fifteenththrough seventeenth frames from the left in the previous frame #n−1contain only background components and belong to the background area.The fifteenth through seventeenth pixels from the left in frame #n−2,which is one before frame #n−1, contain only background components andbelong to the background area.

[0351] Since the object corresponding to the background is stationary,the pixel value of the fifteenth pixel from the left in frame #n−1 doesnot change from the pixel value of the fifteenth pixel from the left inframe #n−2. Similarly, the pixel value of the sixteenth pixel from theleft in frame #n−1 does not change from the pixel value of the sixteenthpixel from the left in frame #n−2, and the pixel value of theseventeenth pixel from the left in frame #n−1 does not change from thepixel value of the seventeenth pixel from the left in frame #n−2.

[0352] That is, the pixels in frame #n−1 and frame #n−2 corresponding tothe pixels belonging to the covered background area in frame #n consistof only background components, and the pixel values thereof do notchange. Accordingly, the absolute value of the difference between thepixel values is almost 0. Thus, the stationary/moving determination madefor the pixels in frame #n−1 and frame #n−2 corresponding to the pixelsbelonging to the mixed area in frame #n by the stationary/movingdetermining portion 202-4 is “stationary”.

[0353] Since the pixels belonging to the covered background area inframe #n contain foreground components, the pixel values thereof aredifferent from those of frame #n−1 consisting of only backgroundcomponents. Accordingly, the stationary/moving determination made forthe pixels belonging to the mixed area in frame #n and the correspondingpixels in frame #n−1 by the stationary/moving determining portion 202-3is “moving”.

[0354] When the stationary/moving determination result indicating“moving” is supplied from the stationary/moving determining portion202-3, and when the stationary/moving determination result indicating“stationary” is supplied from the stationary/moving determining portion202-4, as discussed above, the area determining portion 203-3 determinesthat the corresponding pixels belong to a covered background area.

[0355] In FIG. 24, in frame #n in which the background is stationary andthe amount of movement v in the foreground is 4, the pixels contained inan uncovered background area are the second through fourth pixels fromthe left. Since the amount of movement v is 4, the second through fourthpixels from the left in the subsequent frame #n+1 contain onlybackground components and belong to the background area. In frame #n+2,which is subsequent to frame #n+1, the second through fourth pixels fromthe left contain only background components and belong to the backgroundarea.

[0356] Since the object corresponding to the background is stationary,the pixel value of the second pixel from the left in frame #n+2 does notchange from the pixel value of the second pixel from the left in frame#n+1. Similarly, the pixel value of the third pixel from the left inframe #n+2 does not change from the pixel value of the third pixel fromthe left in frame #n+1, and the pixel value of the fourth pixel from theleft in frame #n+2 does not change from the pixel value of the fourthpixel from the left in frame #n+1.

[0357] That is, the pixels in frame #n+1 and frame #n+2 corresponding tothe pixels belonging to the uncovered background area in frame #nconsist of only background components, and the pixel values thereof donot change. Accordingly, the absolute value of the difference betweenthe pixel values is almost 0. Thus, the stationary/moving determinationmade for the pixels in frame #n+1 and frame #n+2 corresponding to thepixels belonging to the mixed area in frame #n by the stationary/movingdetermining portion 202-1 is “stationary”.

[0358] Since the pixels belonging to the uncovered background area inframe #n contain foreground components, the pixel values thereof aredifferent from those of frame #n+1 consisting of only backgroundcomponents. Accordingly, the stationary/moving determination made forthe pixels belonging to the mixed area in frame #n and the correspondingpixels in frame #n+1 by the stationary/moving determining portion 202-2is “moving”.

[0359] When the stationary/moving determination result indicating“moving” is supplied from the stationary/moving determining portion202-2, and when the stationary/moving determination result indicating“stationary” is supplied from the stationary/moving determining portion202-1, as discussed above, the area determining portion 203-1 determinesthat the corresponding pixels belong to an uncovered background area.

[0360]FIG. 25 illustrates determination conditions for frame #n made bythe area specifying unit 103. When the determination result for thepixel in frame #n−2 located at the same image position as a pixel inframe #n to be processed and for the pixel in frame #n−1 located at thesame position as the pixel in frame #n is stationary, and when thedetermination result for the pixel in frame #n and the pixel in frame#n−1 located at the same image position as the pixel in frame #n ismoving, the area specifying unit 103 determines that the pixel in frame#n belongs to a covered background area.

[0361] When the determination result for the pixel in frame #n and thepixel in frame #n−1 located at the same image position as the pixel inframe #n is stationary, and when the determination result for the pixelin frame #n and the pixel in frame #n+1 located at the same imageposition as the pixel in frame #n is stationary, the area specifyingunit 103 determines that the pixel in frame #n belongs to the stationaryarea.

[0362] When the determination result for the pixel in frame #n and thepixel in frame #n−1 located at the same image position as the pixel inframe #n is moving, and when the determination result for the pixel inframe #n and the pixel in frame #n+1 located at the same image positionas the pixel in frame #n is moving, the area specifying unit 103determines that the pixel in frame #n belongs to the moving area.

[0363] When the determination result for the pixel in frame #n and thepixel in frame #n+1 located at the same image position as the pixel inframe #n is moving, and when the determination result for the pixel inframe #n+1 located at the same image position as the pixel in frame #nand the pixel in frame #n+2 located at the same image position as thepixel in frame #n is stationary, the area specifying unit 103 determinesthat the pixel in frame #n belongs to an uncovered background area.

[0364]FIGS. 26A through 26D illustrate examples of the areadetermination results obtained by the area specifying unit 103. In FIG.26A, the pixels which are determined to belong to a covered backgroundarea are indicated in white. In FIG. 26B, the pixels which aredetermined to belong to an uncovered background area are indicated inwhite.

[0365] In FIG. 26C, the pixels which are determined to belong to amoving area are indicated in white. In FIG. 26D, the pixels which aredetermined to belong to a stationary area are indicated in white.

[0366]FIG. 27 illustrates the area information indicating the mixedarea, in the form of an image, selected from the area information outputfrom the determining-flag-storing frame memory 206. In FIG. 27, thepixels which are determined to belong to the covered background area orthe uncovered background area, i.e., the pixels which are determined tobelong to the mixed area, are indicated in white. The area informationindicating the mixed area output from the determining-flag-storing framememory 206 designates the mixed area and the portions having a texturesurrounded by the portions without a texture in the foreground area.

[0367] The area specifying processing performed by the area specifyingunit 103 is described below with reference to the flowchart of FIG. 28.In step S201, the frame memory 201 obtains an image of frame #n−2through frame #n+2 including frame #n, for which a determination is tobe made.

[0368] In step S202, the stationary/moving determining portion 202-3determines whether the determination result for the pixel in frame #n−1and the pixel in frame #n located at the same position is stationary. Ifit is determined that the determination result is stationary, theprocess proceeds to step S203 in which the stationary/moving determiningportion 202-2 determines whether the determination result for the pixelin frame #n and the pixel in frame #n+1 located at the same position isstationary.

[0369] If it is determined in step S203 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is stationary, the process proceeds to step S204. In stepS204, the area determining portion 203-2 sets “1”, which indicates thatthe pixel to be processed belongs to the stationary area, in thestationary-area determining flag associated with the pixel to beprocessed. The area determining portion 203-2 supplies thestationary-area determining flag to the determining-flag-storing framememory 204, and the process proceeds to step S205.

[0370] If it is determined in step S202 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is moving, or if it is determined in step S203 that thedetermination result for the pixel in frame #n and the pixel in frame#n+1 located at the same position is moving, the pixel to be processeddoes not belong to a stationary area. Accordingly, the processing ofstep S204 is skipped, and the process proceeds to step S205.

[0371] In step S205, the stationary/moving determining portion 202-3determines whether the determination result for the pixel in frame #n−1and the pixel in frame #n located at the same position is moving. If itis determined that the determination result is moving, the processproceeds to step S206 in which the stationary/moving determining portion202-2 determines whether the determination result for the pixel in frame#n and the pixel in frame #n+1 located at the same position is moving.

[0372] If it is determined in step S206 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is moving, the process proceeds to step S207. In stepS207, the area determining portion 203-2 sets “1”, which indicates thatthe pixel to be processed belongs to a moving area, in the moving-areadetermining flag associated with the pixel to be processed. The areadetermining area 203-2 supplies the moving-area determining flag to thedetermining-flag-storing frame memory 204, and the process proceeds tostep S208.

[0373] If it is determined in step S205 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is stationary, or if it is determined in step S206 thatthe determination result for the pixel in frame #n and the pixel inframe #n+1 located at the same position is stationary, the pixel inframe #n does not belong to a moving area. Accordingly, the processingof step S207 is skipped, and the process proceeds to step S208.

[0374] In step S208, the stationary/moving determining portion 202-4determines whether the determination result for the pixel in frame #n−2and the pixel in frame #n−1 located at the same position is stationary.If it is determined that the determination result is stationary, theprocess proceeds to step S209 in which the stationary/moving determiningportion 202-3 determines whether the determination result for the pixelin frame #n−1 and the pixel in frame #n located at the same position ismoving.

[0375] If it is determined in step S209 that the determination resultfor the pixel in frame #n−1 and the pixel in frame #n located at thesame position is moving, the process proceeds to step S210. In stepS210, the area determining portion 203-3 sets “1”, which indicates thatthe pixel to be processed belongs to a covered background area, in thecovered-background-area determining flag associated with the pixel to beprocessed. The area determining portion 203-3 supplies thecovered-background-area determining flag to the determining-flag-storingframe memory 204, and the process proceeds to step S211. The areadetermining portion 203-3 supplies the covered-background-areadetermining flag to the determining-flag-storing frame memory 204, andthe process proceeds to step S211.

[0376] If it is determined in step S208 that the determination resultfor the pixel in frame #n−2 and the pixel in frame #n−1 located at thesame position is moving, or if it is determined in step S209 that thepixel in frame #n−1 and the pixel in frame #n located at the sameposition is stationary, the pixel in frame #n does not belong to acovered background area. Accordingly, the processing of step S210 isskipped, and the process proceeds to step S211.

[0377] In step S211, the stationary/moving determining portion 202-2determines whether the determination result for the pixel in frame #nand the pixel in frame #n+1 located at the same position is moving. Ifit is determined in step S211 that the determination result is moving,the process proceeds to step S212 in which the stationary/movingdetermining portion 202-1 determines whether the determination resultfor the pixel in frame #n+1 and the pixel in frame #n+2 located at thesame position is stationary.

[0378] If it is determined in step S212 that the determination resultfor the pixel in frame #n+1 and the pixel in frame #n+2 located at thesame position is stationary, the process proceeds to step S213. In stepS213, the area determining portion 203-1 sets “1”, which indicates thatthe pixel to be processed belongs to an uncovered background area, inthe uncovered-background-area determining flag associated with the pixelto be processed. The area determining portion 203-1 supplies theuncovered-background-flag determining flag to thedetermining-flag-storing frame memory 204, and the process proceeds tostep S214.

[0379] If it is determined in step S211 that the determination resultfor the pixel in frame #n and the pixel in frame #n+1 located at thesame position is stationary, or if it is determined in step S212 thatthe determination result for the pixel in frame #n+1 and the pixel inframe #n+2 is moving, the pixel in frame #n does not belong to anuncovered background area. Accordingly, the processing of step S213 isskipped, and the process proceeds to step S214.

[0380] In step S214, the area specifying unit 103 determines whether theareas of all the pixels in frame #n are specified. If it is determinedthat the areas of all the pixels in frame #n are not yet specified, theprocess returns to step S202, and the area specifying processing isrepeated for the remaining pixels.

[0381] If it is determined in step S214 that the areas of all the pixelsin frame #n are specified, the process proceeds to step S215. In stepS215, the synthesizer 215 generates area information indicating themixed area based on the uncovered-background-area determining flag andthe covered-background-area determining flag stored in thedetermining-flag-storing frame memory 204, and also generates areainformation indicating to which of the uncovered background area, thestationary area, the moving area, or the covered background area eachpixel belongs, and sets the generated area information in thedetermining-flag-storing frame memory 206. The processing is thencompleted.

[0382] As discussed above, the area specifying unit 103 is capable ofgenerating area information indicating to which of the moving area, thestationary area, the uncovered background area, or the coveredbackground area each of the pixels contained in a frame belongs.

[0383] The area specifying unit 103 may apply logical OR to the areainformation corresponding to the uncovered background area and the areainformation corresponding to the covered background area so as togenerate area information corresponding to the mixed area, and then maygenerate area information consisting of flags indicating to which of themoving area, the stationary area, or the mixed area the individualpixels contained in the frame belong.

[0384] When the object corresponding to the foreground has a texture,the area specifying unit 103 is able to specify the moving area moreprecisely.

[0385] The area specifying unit 103 is able to output the areainformation indicating the moving area as the area informationindicating the foreground area, and outputs the area informationindicating the stationary area as the area information indicating thebackground area.

[0386] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described area specifying processing can be applied even if theimage corresponding to the background area contains motion. For example,if the image corresponding to the background area is uniformly moving,the area specifying unit 103 shifts the overall image in accordance withthis motion, and performs processing in a manner similar to the case inwhich the object corresponding to the background is stationary. If theimage corresponding to the background area contains locally differentmotions, the area specifying unit 103 selects the pixels correspondingto the motions, and executes the above-described processing.

[0387]FIG. 29 is a block diagram illustrating another example of theconfiguration of the area specifying unit 103. The area specifying unit103 shown in FIG. 29 does not use a motion vector. A background imagegenerator 301 generates a background image corresponding to an inputimage, and supplies the generated background image to abinary-object-image extracting portion 302. The background imagegenerator 301 extracts, for example, an image object corresponding to abackground object contained in the input image, and generates thebackground image.

[0388] An example of a model obtained by expanding in the time directionthe pixel values of pixels aligned side-by-side in the moving directionof an image corresponding to a foreground object is shown in FIG. 30.For example, if the moving direction of the image corresponding to theforeground object is horizontal with respect to the screen, the modelshown in FIG. 30 is a model obtained by expanding the pixel values ofpixels disposed side-by-side on a single line in the time direction.

[0389] In FIG. 30, the line in frame #n is the same as the line in frame#n−1 and the line in frame #n+1.

[0390] In frame #n, the foreground components corresponding to theobject contained in the sixth through seventeenth pixels from the leftare contained in the second through thirteenth pixels from the left inframe #n−1 and are also contained in the tenth through twenty-firstpixel from the left in frame #n+1.

[0391] In frame #n−1, the pixels belonging to the covered backgroundarea are the eleventh through thirteenth pixels from the left, and thepixels belonging to the uncovered background area are the second throughfourth pixels from the left. In frame #n, the pixels belonging to thecovered background area are the fifteenth through seventeenth pixelsfrom the left, and the pixels belonging to the uncovered background areaare the sixth through eighth pixels from the left. In frame #n+1, thepixels belonging to the covered background area are the nineteenththrough twenty-first pixels from the left, and the pixels belonging tothe uncovered background area are the tenth through twelfth pixels fromthe left.

[0392] In frame #n−1, the pixels belonging to the background area arethe first pixel from the left, and the fourteenth through twenty-firstpixels from the left. In frame #n, the pixels belonging to thebackground area are the first through fifth pixels from the left, andthe eighteenth through twenty-first pixels from the left. In frame #n+1,the pixels belonging to the background area are the first through ninthpixels from the left.

[0393] An example of the background image corresponding to the exampleshown in FIG. 30 generated by the background image generator 301 isshown in FIG. 31. The background image consists of the pixelscorresponding to the background object, and does not contain imagecomponents corresponding to the foreground object.

[0394] The binary-object-image extracting portion 302 generates a binaryobject image based on the correlation between the background image andthe input image, and supplies the generated binary object image to atime change detector 303.

[0395]FIG. 32 is a block diagram illustrating the configuration of thebinary-object-image extracting portion 302. A correlation-valuecalculator 321 calculates the correlation between the background imagesupplied from the background image generator 301 and the input image soas to generate a correlation value, and supplies the generatedcorrelation value to a threshold-value processor 322.

[0396] The correlation-value calculator 321 applies equation (4) to, forexample, 3×3-background image blocks having X₄ at the center, as shownin FIG. 33A, and to, for example, 3×3-background image blocks having Y₄at the center which corresponds to the background image blocks, as shownin FIG. 33B, thereby calculating a correlation value corresponding toY₄. $\begin{matrix}{{{Correlation}\quad {value}} = \frac{\sum\limits_{i = 0}^{8}{\left( {{X\quad i} - \overset{\_}{X}} \right){\sum\limits_{i = 0}^{8}\left( {{Y\quad i} - \overset{\_}{Y}} \right)}}}{\sqrt{\sum\limits_{i = 0}^{8}{\left( {{X\quad i} - \overset{\_}{X}} \right)^{2} \cdot {\sum\limits_{i = 0}^{8}\left( {{Y\quad i} - \overset{\_}{Y}} \right)^{2}}}}}} & (4) \\{\overset{\_}{X} = \frac{\sum\limits_{i = 0}^{8}{X\quad i}}{9}} & (5) \\{\overset{\_}{Y} = \frac{\sum\limits_{i = 0}^{8}{Y\quad i}}{9}} & (6)\end{matrix}$

[0397] The correlation-value calculator 321 supplies the correlationvalue calculated for each pixel as discussed above to thethreshold-value processor 322.

[0398] Alternatively, the correlation-value calculator 321 may applyequation (7) to, for example, 3×3-background image blocks having X₄ atthe center, as shown in FIG. 34A, and to, for example, 3×3-backgroundimage blocks having Y₄ at the center which corresponds to the backgroundimage blocks, as shown in FIG. 34B, thereby calculating the sum ofabsolute values of differences corresponding to Y₄. $\begin{matrix}{{{Sum}\quad {of}\quad {absolute}\quad {values}\quad {of}\quad {differences}} = {\sum\limits_{i = 0}^{8}{\left( {{X\quad i} - {Y\quad i}} \right)}}} & (7)\end{matrix}$

[0399] The correlation-value calculator 321 supplies the absolute valuesof the differences calculated as described above to the threshold-valueprocessor 322 as the correlation value.

[0400] The threshold-value processor 322 compares the pixel value of thecorrelation image with a threshold value th0. If the correlation valueis smaller than or equal to the threshold value th0, 1 is set in thepixel value of the binary object image. If the correlation value isgreater than the threshold value th0, 0 is set in the pixel value of thebinary object image. The threshold-value processor 322 then outputs thebinary object image whose pixel value is set to 0 or 1. Thethreshold-value processor 322 may store the threshold value th0 thereinin advance, or may use the threshold value th0 input from an externalsource.

[0401]FIG. 35 illustrates the binary object image corresponding to themodel of the input image shown in FIG. 30. In the binary object image, 0is set in the pixel values of the pixels having a higher correlationwith the background image.

[0402]FIG. 36 is a block diagram illustrating the configuration of thetime change detector 303. When determining the area of a pixel in frame#n, a frame memory 341 stores a binary object image of frame #n−1, frame#n, and frame #n+1 supplied from the binary-object-image extractingportion 302.

[0403] An area determining portion 342 determines the area of each pixelin frame #n based on the binary object image of frame #n−1, frame #n,and frame #n+1 so as to generate area information, and outputs thegenerated area information.

[0404]FIG. 37 illustrates the determinations made by the areadetermining portion 342. When the designated pixel of the binary objectimage in frame #n is 0, the area determining portion 342 determines thatthe designated pixel in frame #n belongs to the background area.

[0405] When the designated pixel of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n−1 is 1, and when the corresponding pixel of the binary objectimage in frame #n+1 is 1, the area determining portion 342 determinesthat the designated pixel in frame #n belongs to the foreground area.

[0406] When the designated pixel of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n−1 is 0, the area determining portion 342 determines that thedesignated pixel in frame #n belongs to a covered background area.

[0407] When the designated pixel of the binary object image in frame #nis 1, and when the corresponding pixel of the binary object image inframe #n+1 is 0, the area determining portion 342 determines that thedesignated pixel in frame #n belongs to an uncovered background area.

[0408]FIG. 38 illustrates an example of the determinations made by thetime change detector 303 on the binary object image corresponding to themodel of the input image shown in FIG. 30. The time change detector 303determines that the first through fifth pixels from the left in frame #nbelong to the background area since the corresponding pixels of thebinary object image in frame #n are 0.

[0409] The time change detector 303 determines that the sixth throughninth pixels from the left belong to the uncovered background area sincethe pixels of the binary object image in frame #n are 1, and thecorresponding pixels in frame #n+1 are 0.

[0410] The time change detector 303 determines that the tenth throughthirteenth pixels from the left belong to the foreground area since thepixels of the binary object image in frame #n are 1, the correspondingpixels in frame #n−1 are 1, and the corresponding pixels in frame #n+1are 1.

[0411] The time change detector 303 determines that the fourteenththrough seventeenth pixels from the left belong to the coveredbackground area since the pixels of the binary object image in frame #nare 1, and the corresponding pixels in frame #n−1 are 0.

[0412] The time change detector 303 determines that the eighteenththrough twenty-first pixels from the left belong to the background areasince the corresponding pixels of the binary object image in frame #nare 0.

[0413] The area specifying processing performed by the area specifyingunit 103 is described below with reference to the flowchart of FIG. 39.In step S301, the background image generator 301 of the area specifyingunit 103 extracts, for example, an image object corresponding to abackground object contained in an input image based on the input imageso as to generate a background image, and supplies the generatedbackground image to the binary-object-image extracting portion 302.

[0414] In step S302, the binary-object-image extracting portion 302calculates a correlation value between the input image and thebackground image supplied from the background image generator 301according to, for example, calculation discussed with reference to FIGS.33A and 33B. In step S303, the binary-object-image extracting portion302 computes a binary object image from the correlation value and thethreshold value th0 by, for example, comparing the correlation valuewith the threshold value th0.

[0415] In step S304, the time change detector 303 executes the areadetermining processing, and the processing is completed.

[0416] Details of the area determining processing in step S304 aredescribed below with reference to the flowchart of FIG. 40. In stepS321, the area determining portion 342 of the time change detector 303determines whether the designated pixel in frame #n stored in the framememory 341 is 0. If it is determined that the designated pixel in frame#n is 0, the process proceeds to step S322. In step S322, it isdetermined that the designated pixel in frame #n belongs to thebackground area, and the processing is completed.

[0417] If it is determined in step S321 that the designated pixel inframe #n is 1, the process proceeds to step S323. In step S323, the areadetermining portion 342 of the time change detector 303 determineswhether the designated pixel in frame #n stored in the frame memory 341is 1, and whether the corresponding pixel in frame #n−1 is 0. If it isdetermined that the designated pixel in frame #n is 1 and thecorresponding pixel in frame #n−1 is 0, the process proceeds to stepS324. In step S324, it is determined that the designated pixel in frame#n belongs to the covered background area, and the processing iscompleted.

[0418] If it is determined in step S323 that the designated pixel inframe #n is 0, or that the corresponding pixel in frame #n−1 is 1, theprocess proceeds to step S325. In step S325, the area determiningportion 342 of the time change detector 303 determines whether thedesignated pixel in frame #n stored in the frame memory 341 is 1, andwhether the corresponding pixel in frame #n+1 is 0. If it is determinedthat the designated pixel in frame #n is 1 and the corresponding pixelin frame #n+1 is 0, the process proceeds to step S326. In step S326, itis determined that the designated pixel in frame #n belongs to theuncovered background area, and the processing is completed.

[0419] If it is determined in step S325 that the designated pixel inframe #n is 0, or that the corresponding pixel in frame #n+1 is 1, theprocess proceeds to step S327. In step S327, the area determiningportion 342 of the time change detector 303 determines that thedesignated pixel in frame #n belongs to the foreground area, and theprocessing is completed.

[0420] As discussed above, the area specifying unit 103 is able tospecify, based on the correlation value between the input image and thecorresponding background image, to which of the foreground area, thebackground area, the covered background area, or the uncoveredbackground area each pixel of the input image belongs, and generatesarea information corresponding to the specified result.

[0421]FIG. 41 is a block diagram illustrating another configuration ofthe area specifying unit 103. The area specifying unit 103 shown in FIG.41 uses a motion vector and positional information thereof supplied fromthe motion detector 102. The same elements as those shown in FIG. 29 aredesignated with like reference numerals, and an explanation thereof isthus omitted.

[0422] A robust-processing portion 361 generates a robust binary objectimage based on binary object images of N frames supplied from thebinary-object-image extracting portion 302, and outputs the robustbinary object image to the time change detector 303.

[0423]FIG. 42 is a block diagram illustrating the configuration of therobust-processing portion 361. A motion compensator 381 compensates forthe motion of the binary object images of N frames based on the motionvector and the positional information thereof supplied from the motiondetector 102, and outputs a motion-compensated binary object image to aswitch 382.

[0424] The motion compensation performed by the motion compensator 381is discussed below with reference to examples shown in FIGS. 43 and 44.It is now assumed, for example, that the area in frame #n is to beprocessed. When binary object images of frame #n−1, frame #n, and frame#n+1 shown in FIG. 43 are input, the motion compensator 381 compensatesfor the motion of the binary object image of frame #n−1 and the binaryobject image of frame #n+1, as indicated by the example shown in FIG.44, based on the motion vector supplied from the motion detector 102,and supplies the motion-compensated binary object images to the switch382.

[0425] The switch 382 outputs the motion-compensated binary object imageof the first frame to a frame memory 383-1, and outputs themotion-compensated binary object image of the second frame to a framememory 383-2. Similarly, the switch 382 outputs the motion-compensatedbinary object images of the third through (N−1)-th frame to framememories 383-3 through 383-(N−1), and outputs the motion-compensatedbinary object image of the N-th frame to a frame memory 383-N.

[0426] The frame memory 383-1 stores the motion-compensated binaryobject image of the first frame, and outputs the stored binary objectimage to a weighting portion 384-1. The frame memory 383-2 stores themotion-compensated binary object image of the second frame, and outputsthe stored binary object image to a weighting portion 384-2.

[0427] Similarly, the frame memories 383-3 through 383-(N−1) stores themotion-compensated binary object images of the third through (N−1)-thframes, and outputs the stored binary object images to weightingportions 384-3 through 384-(N−1). The frame memory 383-N stores themotion-compensated binary object image of the N-th frame, and outputsthe stored binary object image to a weighting portion 384-N.

[0428] The weighting portion 384-1 multiplies the pixel value of themotion-compensated binary object image of the first frame supplied fromthe frame memory 383-1 by a predetermined weight w1, and supplies aweighted binary object image to an accumulator 385. The weightingportion 384-2 multiplies the pixel value of the motion-compensatedbinary object image of the second frame supplied from the frame memory383-2 by a predetermined weight w2, and supplies the weighted binaryobject image to the accumulator 385.

[0429] Likewise, the weighting portions 384-3 through 384-(N−1) multiplythe pixel values of the motion-compensated binary object images of thethird through (N−1)-th frames supplied from the frame memories 383-3through 383-(N−1) by predetermined weights w3 through w(N−1), andsupplies the weighted binary object images to the accumulator 385. Theweighting portion 384-N multiplies the pixel value of themotion-compensated binary object image of the N-th frame supplied fromthe frame memory 383-N by a predetermined weight wN, and supplies theweighted binary object image to the accumulator 385.

[0430] The accumulator 385 accumulates the pixel values of themotion-compensated binary object images multiplied by the weights w1through wN of the first through N-th frames, and compares theaccumulated pixel value with the predetermined threshold value th0,thereby generating the binary object image.

[0431] As discussed above, the robust-processing portion 361 generates arobust binary object image from N binary object images, and supplies itto the time change detector 303. Accordingly, the area specifying unit103 configured as shown in FIG. 41 is able to specify the area moreprecisely than that shown in FIG. 29 even if noise is contained in theinput image.

[0432] The area specifying processing performed by the area specifyingunit 103 configured as shown in FIG. 41 is described below withreference to the flowchart of FIG. 45. The processings of step S341through step S343 are similar to those of step S301 through step S303discussed with reference to the flowchart of FIG. 39, and an explanationthereof is thus omitted.

[0433] In step S344, the robust-processing portion 361 performs therobust processing.

[0434] In step S345, the time change detector 303 performs the areadetermining processing, and the processing is completed. Details of theprocessing of step S345 are similar to the processing discussed withreference to the flowchart of FIG. 40, and an explanation thereof isthus omitted.

[0435] Details of the robust processing corresponding to the processingof step S344 in FIG. 45 are given below with reference to the flowchartof FIG. 46. In step S361, the motion compensator 381 performs the motioncompensation of an input binary object image based on the motion vectorand the positional information thereof supplied from the motion detector102. In step S362, one of the frame memories 383-1 through 383-N storesthe corresponding motion-compensated binary object image supplied viathe switch 382.

[0436] In step S363, the robust-processing portion 361 determineswhether N binary object images are stored. If it is determined that Nbinary object images are not stored, the process returns to step S361,and the processing for compensating for the motion of the binary objectimage and the processing for storing the binary object image arerepeated.

[0437] If it is determined in step S363 that N binary object images arestored, the process proceeds to step S364 in which weighting isperformed. In step S364, the weighting portions 384-1 through 384-Nmultiply the corresponding N binary object images by the weights w1through wN.

[0438] In step S365, the accumulator 385 accumulates the N weightedbinary object images.

[0439] In step S366, the accumulator 385 generates a binary object imagefrom the accumulated images by, for example, comparing the accumulatedvalue with a predetermined threshold value th1, and the processing iscompleted.

[0440] As discussed above, the area specifying unit 103 configured asshown in FIG. 41 is able to generate area information based on therobust binary object image.

[0441] As is seen from the foregoing description, the area specifyingunit 103 is able to generate area information indicating to which of themoving area, the stationary area, the uncovered background area, or thecovered background area each pixel contained in a frame belongs.

[0442]FIG. 47 is a block diagram illustrating an example of theconfiguration of the mixture-ratio calculator 104. Anestimated-mixture-ratio processor 401 calculates an estimated mixtureratio for each pixel by calculations corresponding to a model of acovered background area based on the input image, and supplies thecalculated estimated mixture ratio to a mixture-ratio determiningportion 403.

[0443] An estimated-mixture-ratio processor 402 calculates an estimatedmixture ratio for each pixel by calculations corresponding to a model ofan uncovered background area based on the input image, and supplies thecalculated estimated mixture ratio to the mixture-ratio determiningportion 403.

[0444] Since it can be assumed that the object corresponding to theforeground is moving with constant velocity within the shutter time, themixture ratio α of the pixels belonging to a mixed area exhibits thefollowing characteristics. That is, the mixture ratio α linearly changesaccording to the positional change in the pixels. If the positionalchange in the pixels is one-dimensional, a change in the mixture ratio αcan be represented linearly. If the positional change in the pixels istwo-dimensional, a change in the mixture ratio α can be represented on aplane.

[0445] Since the period of one frame is short, it can be assumed thatthe object corresponding to the foreground is a rigid body moving withconstant velocity.

[0446] The gradient of the mixture ratio α is inversely proportional tothe amount of movement v within the shutter time of the foreground.

[0447] An example of the ideal mixture-ratio α is shown in FIG. 48. Thegradient l of the ideal mixture-ratio α in the mixed area can berepresented by the reciprocal of the amount of movement v.

[0448] As shown in FIG. 48, the ideal mixture-ratio α has the value of 1in the background area, the value of 0 in the foreground area, and thevalue of greater than 0 and smaller than 1 in the mixed area.

[0449] In the example shown in FIG. 49, the pixel value C06 of theseventh pixel from the left in frame #n can be indicated by equation (8)by using the pixel value P06 of the seventh pixel from the left in frame#n−1. $\begin{matrix}\begin{matrix}{{C06} = {{{B06}/v} + {{B06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{P06}/v} + {{P06}/v} + {{F01}/v} + {{F02}/v}}} \\{= {{{2/v} \cdot {P06}} + {\sum\limits_{i = 1}^{2}{F\quad {i/v}}}}}\end{matrix} & (8)\end{matrix}$

[0450] In equation (8), the pixel value C06 is indicated by a pixelvalue M of the pixel in the mixed area, while the pixel value P06 isindicated by a pixel value B of the pixel in the background area. Thatis, the pixel value M of the pixel in the mixed area and the pixel valueB of the pixel in the background area can be represented by equations(9) and (10), respectively.

M=C06  (9)

B=P06  (10)

[0451] In equation (8), 2/v corresponds to the mixture ratio α. Sincethe amount of movement v is 4, the mixture ratio α of the seventh pixelfrom the left in frame #n is 0.5.

[0452] As discussed above, the pixel value C in the designated frame #nis considered as the pixel value in the mixed area, while the pixelvalue P of frame #n−1 prior to frame #n is considered as the pixel valuein the background area. Accordingly, equation (3) indicating the mixtureratio α can be represented by equation (11):

C=α·P+f  (11)

[0453] where f in equation (11) indicates the sum of the foregroundcomponents Σ_(i)Fi/v contained in the designated pixel. The variablescontained in equation (11) are two factors, i.e., the mixture ratio αand the sum f of the foreground components.

[0454] Similarly, a model obtained by expanding in the time directionthe pixel values in which the amount of movement is 4 and the number ofvirtual divided portions is 4 in an uncovered background area is shownin FIG. 50.

[0455] As in the representation of the covered background area, in theuncovered background area, the pixel value C of the designated frame #nis considered as the pixel value in the mixed area, while the pixelvalue N of frame #n+1 subsequent to frame #n is considered as thebackground area. Accordingly, equation (3) indicating the mixture ratioα can be represented by equation (12).

C=α·N+f  (12)

[0456] The embodiment has been described, assuming that the backgroundobject is stationary. However, equations (8) through (12) can be appliedto the case in which the background object is moving by using the pixelvalue of a pixel located corresponding to the amount of movement v ofthe background. It is now assumed, for example, in FIG. 49, that theamount of movement v of the object corresponding to the background is 2,and the number of virtual divided portions is 2. In this case, when theobject corresponding to the background is moving to the right in FIG.49, the pixel value B of the pixel in the background area in equation(10) is represented by a pixel value P04.

[0457] Since equations (11) and (12) each contain two variables, themixture ratio α cannot be determined without modifying the equations.Here, generally, since an image has a strong correlation in relation tospace, pixels in proximity to each other have approximately the samepixel values.

[0458] Accordingly, since the foreground components have a strongcorrelation in relation to space, the equation is modified so that thesum f of the foreground components can be derived from the previous orsubsequent frame, and the mixture ratio α is determined.

[0459] The pixel value Mc of the seventh pixel from the left in frame #nin FIG. 51 can be expressed by equation (13): $\begin{matrix}{{M\quad c} = {{\frac{2}{v} \cdot {B06}} + {\sum\limits_{i = 11}^{12}{F\quad {i/v}}}}} & (13)\end{matrix}$

[0460] In equation (13), 2/v of the first term of the right sidecorresponds to the mixture ratio α. The second term of the right side inequation (13) is shown as in equation (14): $\begin{matrix}{{\sum\limits_{i = 11}^{12}{F\quad {i/v}}} = {\beta \cdot {\sum\limits_{i = 7}^{10}{F\quad {i/v}}}}} & (14)\end{matrix}$

[0461] Here, by using the space correlation of the foregroundcomponents, it is assumed that equation (15) holds:

F=F05=F06=F07=F08=F09=F10=F11=F12  (15)

[0462] Equation (14) can be replaced as shown in equation (16) by usingequation (15): $\begin{matrix}\begin{matrix}{{\sum\limits_{i = 11}^{12}{F\quad {i/v}}} = {\frac{2}{v} \cdot F}} \\{= {\beta \cdot \frac{4}{v} \cdot F}}\end{matrix} & (16)\end{matrix}$

[0463] As a result, β can be expressed by equation (17):

β=2/4  (17)

[0464] In general, if it is assumed that, as shown in equation (15), theforeground components related to the mixed area are equal, equation (18)holds for all the pixels of the mixed area on the basis of the relationof the internal ratio:

β=1−α  (18)

[0465] If it is assumed that equation (18) holds, equation (11) can beexpanded as shown in equation (19): $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot P} + f}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{F\quad {i/v}}}}}} \\{= {{\alpha \cdot P} + {\left( {1 - \alpha} \right) \cdot N}}}\end{matrix} & (19)\end{matrix}$

[0466] Similarly, if it is assumed that equation (18) holds, equation(12) can be expanded as shown in equation (20): $\begin{matrix}\begin{matrix}{C = {{\alpha \cdot N} + f}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot {\sum\limits_{i = \gamma}^{\gamma + V - 1}{F\quad {i/v}}}}}} \\{= {{\alpha \cdot N} + {\left( {1 - \alpha} \right) \cdot P}}}\end{matrix} & (20)\end{matrix}$

[0467] In equations (19) and (20), since C, N, and P are known pixelvalues, the variables contained in equations (19) and (20) are only themixture ratio α. The relationship among C, N, and P in equations (19)and (20) is shown in FIG. 52. C indicates the pixel value of thedesignated pixel in frame #n. N indicates the pixel value of the pixelin frame #n+1, whose position in the spatial direction corresponds tothe designated pixel. P indicates the pixel value of the pixel in frame#n−1, whose position in the spatial direction corresponds to thedesignated pixel.

[0468] Therefore, since equations (19) and (20) each contain a singlevariable, the mixture ratio α can be calculated using the pixel valuesof the pixels of three frames. By solving equations (19) and (20), it isshown that the condition under which the correct mixture ratio α iscalculated is that the foreground components related to the mixed areaare equal, that is, the pixel values of the successive pixels of anumber twice the amount of movement v, which are the pixels positionedat the boundary of the image object, corresponding to the movingdirection of the object of the foreground, in the image object of theforeground, which is captured when the foreground object is stationary,are fixed.

[0469] As discussed above, the mixture ratio α of the pixels belongingto the covered background area is calculated on the basis of equation(21), and the mixture ratio α of the pixels belonging to the uncoveredbackground area is calculated on the basis of equation (22).

α=(C−N)/(P−N)  (21)

α=(C−P)/(N−P)  (22)

[0470]FIG. 53 is a block diagram illustrating the configuration of themixture ratio estimation processor 401. A frame memory 421 stores inputimages in units of frames, and supplies the frame which is one frameafter the frame input as the input image to a frame memory 422 and amixture-ratio calculator 423.

[0471] The frame memory 422 stores input images in units of frames, andsupplies the frame which is one frame after the frame supplied from theframe memory 421 to the mixture-ratio calculator 423.

[0472] Therefore, when the frame #n+1 has been input as an input imageto the mixture-ratio calculator 423, the frame memory 421 supplies frame#n to the mixture-ratio calculator 423, and the frame memory 422supplies the frame #n−1 to the mixture-ratio calculator 423.

[0473] Based on the calculation shown in equation (21), themixture-ratio calculator 423 calculates the estimated mixture ratio ofthe designated pixel, on the basis of the pixel value C of thedesignated pixel in frame #n, the pixel value N of the pixel in frame#n+1, whose spatial position corresponds to the designated pixel, andthe pixel value P of the pixel in frame #n−1, whose spatial positioncorresponds to the designated pixel, and outputs the calculatedestimated mixture ratio. For example, when the background is stationary,the mixture-ratio calculator 423 calculates the estimated mixture ratioof the designated pixel on the basis of the pixel value C of thedesignated pixel in frame #n, the pixel value N of the pixel in frame#n+1, whose position within the frame is the same as that of thedesignated pixel, and the pixel value P of the pixel in frame #n−1,whose position within the frame is the same as that of the designatedpixel, and outputs the calculated estimated mixture ratio.

[0474] In this manner, the estimated-mixture-ratio processor 401 is ableto calculate the estimated mixture ratio based on the input image, andsupplies it to the mixture-ratio determining portion 403.

[0475] The estimated-mixture-ratio processor 402 is configured the sameas the estimated-mixture-ratio processor 401 except that, whereas theestimated-mixture-ratio processor 401 calculates the estimated mixtureratio of the designated pixel on the basis of the calculation shown inequation (21), the estimated-mixture-ratio processor 402 calculates theestimated mixture ratio of the designated pixel on the basis of thecalculation shown in equation (22), and an explanation thereof is thusomitted.

[0476]FIG. 54 shows an example of an estimated mixture ratio calculatedby the estimated-mixture-ratio processor 401. The estimated mixtureratio shown in FIG. 54 shows, for one line, the result in a case wherethe amount of movement v of the foreground corresponding to an objectmoving with constant velocity is 11.

[0477] It can be seen that the estimated mixture ratio nearly changeslinearly, as shown in FIG. 48.

[0478] Referring back to FIG. 47, a mixture-ratio determining portion403 sets the mixture ratio α based on the area information supplied fromthe area specifying unit 103 and indicating to which of the foregroundarea, the background area, the covered background area, or the uncoveredbackground area the pixel for which the mixture ratio α is to becalculated belongs. The mixture-ratio determining portion 403 sets themixture ratio α to 0 when the corresponding pixel belongs to theforeground area, and sets the mixture ratio α to 1 when thecorresponding pixel belongs to the background area. When thecorresponding pixel belongs to the covered background area, themixture-ratio determining portion 403 sets the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 401 as the mixtureratio α. When the corresponding pixel belongs to the uncoveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 402 as the mixture ratio α. The mixture-ratio determiningportion 403 outputs the mixture ratio α which has been set based on thearea information.

[0479]FIG. 55 is a block diagram illustrating another configuration ofthe mixture-ratio calculator 104. A selector 441 supplies a pixelbelonging to the covered background area and the corresponding pixel inthe previous and subsequent frames to an estimated-mixture-ratioprocessor 442 based on the area information supplied from the areaspecifying unit 103. The selector 441 supplies a pixel belonging to theuncovered background area and the corresponding pixel in the previousand subsequent frames to an estimated-mixture-ratio processor 443 basedon the area information supplied from the area specifying unit 103.

[0480] Based on the pixel values input from the selector 441, theestimated-mixture-ratio processor 442 calculates the estimated mixtureratio of the designated pixel belonging to the covered background areaby the calculation shown in equation (21), and supplies the calculatedestimated mixture ratio to a selector 444.

[0481] Based on the pixel values input from the selector 441, theestimated-mixture-ratio processor 443 calculates the estimated mixtureratio of the designated pixel belonging to the uncovered background areaby the calculation shown in equation (22), and supplies the calculatedestimated mixture ratio to the selector 444.

[0482] Based on the area information supplied from the area specifyingunit 103, the selector 444 sets the mixture ratio α to 0 when thedesignated pixel belongs to the foreground area, and sets the mixtureratio α to 1 when the designated pixel belongs to the background area.When the designated pixel belongs to the covered background area, theselector 444 selects the estimated mixture ratio supplied from theestimated-mixture-ratio processor 442 and sets it as the mixture ratioα. When the designated pixel belongs to the uncovered background area,the selector 444 selects the estimated mixture ratio supplied from theestimated-mixture-ratio processor 443 and sets it as the mixture ratioα. The selector 444 then outputs the mixture ratio α which has beenselected and set based on the area information.

[0483] As discussed above, the mixture-ratio calculator 104 configuredas shown in FIG. 55 is able to calculate the mixture ratio α for eachpixel contained in the image, and outputs the calculated mixture ratioα.

[0484] The calculation processing for the mixture ratio α performed bythe mixture-ratio calculator 104 configured as shown in FIG. 47 isdiscussed below with reference to the flowchart of FIG. 56. In stepS401, the mixture-ratio calculator 104 obtains area information suppliedfrom the area specifying unit 103. In step S402, theestimated-mixture-ratio processor 401 executes the processing forestimating the mixture ratio by using a model corresponding to a coveredbackground area, and supplies the estimated mixture ratio to themixture-ratio determining portion 403. Details of the processing forestimating the mixture ratio are discussed below with reference to theflowchart of FIG. 57.

[0485] In step S403, the estimated-mixture-ratio processor 402 executesthe processing for estimating the mixture ratio by using a modelcorresponding to an uncovered background area, and supplies theestimated mixture ratio to the mixture-ratio determining portion 403.

[0486] In step S404, the mixture-ratio calculator 104 determines whetherthe mixture ratios have been estimated for the whole frame. If it isdetermined that the mixture ratios have not yet been estimated for thewhole frame, the process returns to step S402, and the processing forestimating the mixture ratio α for the subsequent pixel is executed.

[0487] If it is determined in step S404 that the mixture ratios havebeen estimated for the whole frame, the process proceeds to step S405.In step S405, the mixture-ratio determining portion 403 sets the mixtureratio based on the area information supplied from the area specifyingunit 103 and indicating to which of the foreground area, the backgroundarea, the covered background area, or the uncovered background area thepixel for which the mixture ratio α is to be calculated belongs. Themixture-ratio determining portion 403 sets the mixture ratio α to 0 whenthe corresponding pixel belongs to the foreground area, and sets themixture ratio α to 1 when the corresponding pixel belongs to thebackground area. When the corresponding pixel belongs to the coveredbackground area, the mixture-ratio determining portion 403 sets theestimated mixture ratio supplied from the estimated-mixture-ratioprocessor 401 as the mixture ratio α. When the corresponding pixelbelongs to the uncovered background area, the mixture-ratio determiningportion 403 sets the estimated mixture ratio supplied from theestimated-mixture-ratio processor 402 as the mixture ratio α. Theprocessing is then completed.

[0488] As discussed above, the mixture-ratio calculator 104 is able tocalculate the mixture ratio α, which indicates a feature quantitycorresponding to each pixel, based on the area information supplied fromthe area specifying unit 103, and the input image.

[0489] The processing for calculating the mixture ratio α performed bythe mixture-ratio calculator 104 configured as shown in FIG. 55 issimilar to that discussed with reference to the flowchart of FIG. 56,and an explanation thereof is thus omitted.

[0490] A description is now given, with reference to the flowchart ofFIG. 57, of the mixture-ratio estimating processing by using a model ofthe covered background area in step S402 of FIG. 56.

[0491] In step S421, the mixture-ratio calculator 423 obtains the pixelvalue C of the designated pixel in frame #n from the frame memory 421.

[0492] In step S422, the mixture-ratio calculator 423 obtains the pixelvalue P of the pixel in frame #n−1, corresponding to the designatedpixel, from the frame memory 422.

[0493] In step S423, the mixture-ratio calculator 423 obtains the pixelvalue N of the pixel in frame #n+1, corresponding to the designatedpixel contained in the input image.

[0494] In step S424, the mixture-ratio calculator 423 calculates theestimated mixture ratio on the basis of the pixel value C of thedesignated pixel in frame #n, the pixel value P of the pixel in frame#n−1, and the pixel value N of the pixel in frame #n+1.

[0495] In step S425, the mixture-ratio calculator 423 determines whetheror not processing for calculating the estimated mixture ratio isterminated for the whole frame. When it is determined that processingfor calculating the estimated mixture ratio is not terminated for thewhole frame, the process returns to step S421, and the processing forcalculating the estimated mixture ratio for the next pixel is repeated.

[0496] When it is determined in step S425 that processing forcalculating the estimated mixture ratio is terminated for the wholeframe, the processing is terminated.

[0497] As discussed above, the estimated-mixture-ratio processor 401 isable to calculate the estimated mixture ratio based on the input image.

[0498] The mixture-ratio estimating processing by using a modelcorresponding to the uncovered background area in step S403 of FIG. 56is similar to the processing indicated by the flowchart of FIG. 57 byusing the equations corresponding to a model of the uncovered backgroundarea, and an explanation thereof is thus omitted.

[0499] The estimated-mixture-ratio processor 442 and theestimated-mixture-ratio processor 443 shown in FIG. 55 performprocessing similar to that of the flowchart shown in FIG. 57 in order tocalculate the estimated mixture ratio, and an explanation thereof isthus omitted.

[0500] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, processing fordetermining the above-described mixture-ratio α can be applied even ifthe image corresponding to the background area contains motion. Forexample, if the image corresponding to the background area is uniformlymoving, the estimated-mixture-ratio processor 401 shifts the overallimage in accordance with the motion of the background, and performsprocessing in a manner similar to the case in which the objectcorresponding to the background is stationary. If the imagecorresponding to the background area contains locally different motionsof the background, the estimated-mixture-ratio processor 401 selects thepixels corresponding to the motions of the background as the pixelsbelonging to the mixed area, and executes the above-describedprocessing.

[0501] The mixture-ratio calculator 104 may execute the mixture-ratioestimating processing on all the pixels only by using a modelcorresponding to the covered background area, and outputs the calculatedestimated mixture ratio as the mixture ratio α. In this case, themixture ratio α indicates the ratio of the background components for thepixels belonging to the covered background area, and indicates the ratioof the foreground components for the pixels belonging to the uncoveredbackground area. Concerning the pixels belonging to the uncoveredbackground area, the absolute value of the difference between thecalculated mixture-ratio α and 1 is determined, and the calculatedabsolute value is set as the mixture ratio α. Then, the image processingapparatus is able to determine the mixture ratio α indicating the ratioof the background components for the pixels belonging to the uncoveredbackground area.

[0502] Similarly, the mixture-ratio processor 104 may execute themixture-ratio estimating processing on all the pixels only by using amodel corresponding to the uncovered background area, and outputs thecalculated estimated mixture ratio as the mixture ratio α.

[0503] Next, a description is given of the estimated-mixture-ratiocalculator 104 for calculating the mixture ratio α by usingcharacteristics in which the mixture ratio α changes linearly.

[0504] As discussed above, since equations (11) and (12) each containtwo variables, the mixture ratio α cannot be determined withoutmodifying the equations.

[0505] By utilizing the characteristics in which the mixture ratio αlinearly changes in accordance with a change in the position of thepixels because the object corresponding to the foreground is moving withconstant velocity, an equation in which the mixture ratio α and the sumf of the foreground components are approximated in the spatial directionis established. By utilizing a plurality of sets of the pixel values ofthe pixels belonging to the mixed area and the pixel values of thepixels belonging to the background area, the equations in which themixture ratio α and the sum f of the foreground components areapproximated are solved.

[0506] When a change in the mixture ratio α is approximated as astraight line, the mixture ratio α can be expressed by equation (23).

α=il+p  (23)

[0507] In equation (23), i indicates the spatial index when the positionof the designated pixel is set to 0, l designates the gradient of thestraight line of the mixture ratio α, and p designates the intercept ofthe straight line of the mixture ratio α and also indicates the mixtureratio α of the designated pixel. In equation (23), the index i is known,and the gradient l and the intercept p are unknown.

[0508] The relationship among the index i, the gradient l, and theintercept p is shown in FIG. 58. In FIG. 58, the while dot indicates thedesignated pixel, and the black dots indicate the pixels located inclose proximity with the designated pixel.

[0509] By approximating the mixture ratio α as equation (23), aplurality of different mixture-ratios α for a plurality of pixels can beexpressed by two variables. In the example shown in FIG. 58, the fivemixture-ratios for five pixels are expressed by the two variables, i.e.,the gradient l and the intercept p.

[0510] When the mixture ratio α is approximated in the plane shown inFIG. 59, equation (23) is expanded into the plane by considering themovement v corresponding to the two directions, i.e., the horizontaldirection and the vertical direction of the image, and the mixture ratioα can be expressed by equation (24). In FIG. 59, the white dot indicatesthe designated pixel.

α=jm+kq+p  (24)

[0511] In equation (24), j is the index in the horizontal direction andk is the index in the vertical direction when the position of thedesignated pixel is 0. m designates the horizontal gradient of themixture ratio α in the plane, and q indicates the vertical gradient ofthe mixture ratio α in the plane. p indicates the intercept of themixture ratio α in the plane.

[0512] For example, in frame #n shown in FIG. 49, equations (25) through(27) can hold true for C05 through C07, respectively.

C05=α05·B05/v+f05  (25)

C06=α06·B06/v+f06  (26)

C07=α07·B07/v+f07  (27)

[0513] Assuming that the foreground components positioned in closeproximity with each other are equal to each other, i.e., that F01through F03 are equal, equation (28) holds true by replacing F01 throughF03 by fc.

f(x)=(1−α(x))·Fc  (28)

[0514] In equation (28), x indicates the position in the spatialdirection.

[0515] When α(x) is replaced by equation (24), equation (28) can beexpressed by equation (29). $\begin{matrix}\begin{matrix}{{f(x)} = {{\left( {1 - \left( {{j\quad m} + {k\quad q} + p} \right)} \right) \cdot F}\quad c}} \\{= {{j \cdot \left( {{{- m} \cdot F}\quad c} \right)} + {k \cdot \left( {{{- q} \cdot F}\quad c} \right)} + \left( {{\left( {1 - p} \right) \cdot F}\quad c} \right)}} \\{= {{j\quad s} + {k\quad t} + u}}\end{matrix} & (29)\end{matrix}$

[0516] In equation (29), (−m·Fc), (−q·Fc), and (1−p)·Fc are replaced, asexpressed by equations (30) through (32), respectively.

s=−m·Fc  (30)

t=−q·Fc  (31)

u=(1−p)·Fc  (32)

[0517] In equation (29), j is the index in the horizontal direction andk is the index in the vertical direction when the position of thedesignated pixel is 0.

[0518] As discussed above, since it can be assumed that the objectcorresponding to the foreground is moving with constant velocity withinthe shutter time, and that the foreground components positioned in closeproximity with each other are uniform, the sum of the foregroundcomponents can be approximated by equation (29).

[0519] When the mixture ratio α is approximated by a straight line, thesum of the foreground components can be expressed by equation (33).

f(x)=is+u  (33)

[0520] By replacing the mixture ratio α and the sum of the foregroundcomponents in equation (92) by using equations (24) and (29), the pixelvalue M can be expressed by equation (34). $\begin{matrix}\begin{matrix}{M = {{\left( {{j\quad m} + {k\quad q} + p} \right) \cdot B} + {j\quad s} + {k\quad t} + u}} \\{= {{j\quad {B \cdot m}} + {k\quad {B \cdot q}} + {B \cdot p} + {j \cdot s} + {k \cdot t} + u}}\end{matrix} & (34)\end{matrix}$

[0521] In equation (34), unknown variables are six factors, such as thehorizontal gradient m of the mixture ratio α in the plane, the verticalgradient q of the mixture ratio α in the plane, and the intercepts ofthe mixture ratio α in the plane, p, s, t, and u.

[0522] According to the pixels in close proximity with the designatedpixel, the pixel value M or the pixel value B is set in the normalequation shown in equation (34). Then, a plurality of normal equationsin which the pixel value M or the pixel value B is set are solved by themethod of least squares, thereby calculating the mixture ratio α.

[0523] For example, the horizontal index j of the designated pixel isset to 0, and the vertical index k is set to 0. Then, the pixel value Mor the pixel value B is set in normal equation (34) for 3×3 pixelslocated close to the designated pixel, thereby obtaining equations (35)through (43).

M _(−1,−1)=(−1)·B _(−1,−1) ·m+(−1)·B _(−1,−1) ·q+B _(−1,−1)·p+(−1)·s+(−1)·t+u  (35)

M _(0,−1)=(0)·B _(0,−1) ·m+(−1)·B _(0,−1) ·q+B _(0,−1)·p+(0)·s+(−1)·t+u  (36)

M _(+1,−1)=(+1)·B _(+1,−1) ·m+(−1)·B _(+1,−1) ·q+B _(+1,−1)·p+(+1)·s+(−1)·t+u  (37)

M _(−1,0)=(−1)·B _(−1,0) ·m+(0)·B _(−1,0) ·q+B _(1,0)·p+(−1)·s+(0)·t+u  (38)

M _(0,0)=(0)·B _(0,0) ·m+(0)·B _(0,0) ·q+B _(0,0) ·p+(0)·s+(0)·t+u  (39)

M _(+1,0)=(+1)·B _(+1,0) ·m+(0)·B _(+1,0) ·q+B _(+1,0)·p+(+1)·s+(0)·t+u  (40)

M _(−1,+1)=(−1)·B _(−,+1) ·m+(+1)·B _(−1,+1) ·q+B _(−1,+1)·p+(−1)·s+(+1)·t+u  (41)

M _(0,+1)=(0)·B _(0,+1) ·m+(+1)·B _(0,+1) ·q+B _(0,+1)·p+(0)·s+(+1)·t+u  (42)

M _(+1,+1)=(+1)·B _(+1,+1) ·m+(+1)·B _(+1,+1) ·q+B _(+1,+1)·p+(+1)·s+(+1)·t+u  (43)

[0524] Since the horizontal index j of the designated pixel is 0, andthe vertical index k of the designated pixel is 0, the mixture ratio αof the designated pixel is equal to the value when j is 0 and k is 0 inequation (24), i.e., the mixture ratio α is equal to the intercept p inequation (24).

[0525] Accordingly, based on nine equations, i.e., equations (35)through (43), the horizontal gradient m, the vertical gradient q, andthe intercepts p, s, t, and u are calculated by the method of leastsquares, and the intercept p is output as the mixture ratio α.

[0526] A specific process for calculating the mixture ratio α byapplying the method of least squares is as follows.

[0527] When the index i and the index k are indicated by a single indexx, the relationship among the index i, the index k, and the index x isexpressed by equation (44).

x=(j+1)·3+(k+1)  (44)

[0528] It is now assumed that the horizontal gradient m, the verticalgradient q, and the intercepts p, s, t, and u are expressed by variablesw0, w1, w2, w3, w4, and w5, respectively, and jB, kB, B, j, k and l areexpressed by a0, a1, a2, a3, a4, and a5, respectively. In considerationof the error ex, equations (35) through (43) can be modified intoequation (45). $\begin{matrix}{{M\quad x} = {{\sum\limits_{y = 0}^{5}\quad {a\quad {y \cdot w}\quad y}} + {e\quad x}}} & (45)\end{matrix}$

[0529] In equation (45), x is any one of the integers from 0 to 8.

[0530] Equation (46) can be found from equation (45). $\begin{matrix}{{e\quad x} = {{M\quad x} - {\sum\limits_{y = 0}^{5}\quad {a\quad {y \cdot w}\quad y}}}} & (46)\end{matrix}$

[0531] Since the method of least squares is applied, the square sum E ofthe error is defined as follows, as expressed by equation (47).$\begin{matrix}{E = {\sum\limits_{x = 0}^{8}\quad {e\quad x^{2}}}} & (47)\end{matrix}$

[0532] In order to minimize the error, the partial differential value ofthe variable Wv with respect to the square sum E of the error should be0. v is any one of the integers from 0 to 5. Thus, wy is determined sothat equation (48) is satisfied. $\begin{matrix}\begin{matrix}{\frac{\partial E}{{\partial W}\quad v} = {2 \cdot {\sum\limits_{x = 0}^{8}\quad {e\quad {x \cdot \frac{{\partial e}\quad x}{{\partial W}\quad v}}}}}} \\{= {{2 \cdot {\sum\limits_{x = 0}^{8}\quad {e\quad {x \cdot a}\quad v}}} = 0}}\end{matrix} & (48)\end{matrix}$

[0533] By substituting equation (46) into equation (48), equation (49)is obtained. $\begin{matrix}{{\sum\limits_{x = 0}^{8}\left( {a\quad {v \cdot {\sum\limits_{y = 0}^{5}\quad {a\quad {y \cdot W}\quad y}}}} \right)} = {\sum\limits_{x = 0}^{8}\quad {a\quad {v \cdot M}\quad x}}} & (49)\end{matrix}$

[0534] For example, the sweep-out method (Gauss-Jordan elimination) isapplied to the normal equations of six normal equations obtained bysubstituting one of the integers from 0 to 5 into v in equation (49),thereby obtaining wy. As stated above, w0 is the horizontal gradient m,w1 is the vertical gradient q, w2 is the intercept p, w3 is s, w4 is t,and w5 is u.

[0535] As discussed above, by applying the method of least squares tothe equations in which the pixel value M and the pixel value B are set,the horizontal gradient m, the vertical gradient q, and the interceptsp, s, t, and u can be determined.

[0536] A description has been given with reference to equations (35)through (43), by assuming that the pixel value of the pixel contained inthe mixed area is M, and the pixel value of the pixel contained in thebackground area is B. In this case, it is necessary to set normalequations for each of the cases where the designated pixel is containedin the covered background area, or the designated pixel is contained inthe uncovered background area.

[0537] For example, when the mixture ratio α of the pixel contained inthe covered background area in frame #n shown in FIG. 49 is determined,C04 through C08 of the pixels in frame #n and the pixel values P04through P08 of the pixels in frame #n−1 are set in the normal equations.

[0538] For determining the mixture ratio α of the pixel contained in theuncovered background area in frame #n shown in FIG. 50, the pixels C28through C32 of frame #n and the pixel values N28 through N32 of thepixels in frame #n+1 are set in the normal equations.

[0539] Moreover, if, for example, the mixture ratio α of the pixelcontained in the covered background area shown in FIG. 60 is calculated,the following equations (50) through (58) are set. In FIG. 60, the whitedots indicate pixels to belong to the background, and the black dotsindicate pixels to belong to the mixed area. The pixel value of thepixel for which the mixture ratio α is calculated is Mc5.

Mc1=(−1)·Bc1·m+(−1)·Bc1·q+Bc1·p+(−1)·s+(−1)·t+u  (50)

Mc2=(0)·Bc2·m+(−1)·Bc2·q+Bc2·p+(0)·s+(−1)·t+u  (51)

Mc3=(+1)·Bc3·m+(−1)·Bc3·q+Bc3·p+(+1)·s+(−1)·t+u  (52)

Mc4=(−1)·Bc4·m+(0)·Bc4·q+Bc4·p+(−1)·s+(0)·t+u  (53)

Mc5=(0)·Bc5·m+(0)·Bc5·q+Bc5·p+(0)·s+(0)·t+u  (54)

Mc6=(+1)·Bc6·m+(0)·Bc6·q+Bc6·p+(+1)·s+(0)·t+u  (55)

Mc7=(−1)·Bc7·m+(+1)·Bc7·q+Bc7·p+(−1)·s+(+1)·t+u  (56)

Mc8=(0)·Bc8·m+(+1)·Bc8·q+Bc8·p+(0)·s+(+1)·t+u  (57)

Mc9=(+1)·Bc9·m+(+1)·Bc9·q+Bc9·p+(+1)·s+(+1)·t+u  (58)

[0540] For calculating the mixture ratio α of the pixel contained in thecovered background area in frame #n, the pixel values Bc1 through Bc9 ofthe pixels of the background area in frame #n−1 corresponding to thepixels in frame #n in equations (50) through (58), respectively, areused.

[0541] When the mixture ratio α of the pixel contained in the uncoveredbackground area shown in FIG. 60 is calculated, the following equations(59) through (77) are set. The pixel value of the pixel for which themixture ratio α is calculated is Mu5.

Mu1=(−1)·Bu1·m+(−1)·Bu1·q+Bu1·p+(−1)·s+(−1)·t+u  (59)

Mu2=(0)·Bu2·m+(−1)·Bu2·q+Bu2·p+(0)·s+(−1)·t+u  (60)

Mu3=(+1)·Bu3·m+(−1)·Bu3·q+Bu3·p+(+1)·s+(−1)·t+u  (61)

Mu4=(−1)·Bu4·m+(0)·Bu4·q+Bu4·p+(−1)·s+(0)·t+u  (62)

Mu5=(0)·Bu5·m+(0)·Bu5·q+Bu5·p+(0)·s+(0)·t+u  (63)

Mu6=(+1)·Bu6·m+(0)·Bu6·q+Bu6·p+(+1)·s+(0)·t+u  (64)

Mu7=(−1)·Bu7·m+(+1)·Bu7·q+Bu7·p+(−1)·s+(+1)·t+u  (65)

Mu8=(0)·Bu8·m+(+1)·Bu8·q+Bu8·p+(0)·s+(+1)·t+u  (66)

Mu9=(+1)·Bu9·m+(+1)·Bu9·q+Bu9·p+(+1)·s+(+1)·t+u  (67)

[0542] For calculating the mixture ratio α of the pixel contained in theuncovered background area in frame #n, the pixel values Bu1 through Bu9of the pixels of the background area in frame #n+1 corresponding to thepixels in frame #n in equations (59) through (67), respectively, areused.

[0543]FIG. 61 is a block diagram illustrating the configuration of theestimated-mixture-ratio processor 401. An image input into theestimated-mixture-ratio processor 401 is supplied to a delay circuit 501and an adder 502.

[0544] The delay circuit 501 delays the input image for one frame, andsupplies the image to the adder 502. When frame #n is supplied as theinput image to the adder 502, the delay circuit 501 supplies frame #n−1to the adder 502.

[0545] The adder 502 sets the pixel value of the pixel adjacent to thepixel for which the mixture ratio α is calculated, and the pixel valueof frame #n−1 in the normal equation. For example, the adder 502 setsthe pixel values Mc1 through Mc9 and the pixel values Bc1 through Bc9 inthe normal equations based on equations (50) through (58), respectively.The adder 502 supplies the normal equations in which the pixel valuesare set to a calculator 503.

[0546] The calculator 423 determines the estimated mixture ratio bysolving the normal equations supplied from the adder 502 by using thesweep-out method, and outputs the determined estimated mixture ratio.

[0547] In this manner, the estimated-mixture-ratio processor 401 is ableto calculate the estimated mixture ratio based on the input image, andsupplies it to the mixture ratio determining portion 403.

[0548] The estimated-mixture-ratio processor 402 is configured similarto the estimated-mixture-ratio processor 401, and an explanation thereofis thus omitted.

[0549]FIG. 62 shows an example of the estimated mixture ratio calculatedby the estimated-mixture-ratio processor 401. For the estimated mixtureratio shown in FIG. 62, the motion v of the foreground corresponding toan object moving with constant velocity is 11, and the resultscalculated by generating an equation using blocks of 7×7 pixels as unitsare shown for one line.

[0550] It can be seen from FIG. 61 that the estimated mixture ratiochanges approximately linearly in the mixed area.

[0551] A description is now given, with reference to the flowchart ofFIG. 63, of the mixture ratio estimating processing by theestimated-mixture-ratio processor 401 having the configuration shown inFIG. 61 by using a model of the covered background area.

[0552] In step S521, the adder 502 sets the pixel value contained in theinput image and the pixel value contained in the image supplied from thedelay circuit 501 in a normal equation corresponding to a model of thecovered background area.

[0553] In step S522, the estimated-mixture-ratio processor 401determines whether the setting of the target pixels is finished. If itis determined that the setting of the target pixels is not finished, theprocess returns to step S521, and the processing for setting the pixelvalues in the normal equation is repeated.

[0554] If it is determined in step S522 that the setting for the targetpixels is finished, the process proceeds to step S523. In step S523, acalculator 503 calculates the estimated mixture ratio based on thenormal equations in which the pixels values are set, and outputs thecalculated mixture-ratio.

[0555] As discussed above, the estimated-mixture-ratio processor 401having the configuration shown in FIG. 61 is able to calculate theestimated mixture ratio based on the input image.

[0556] The mixture-ratio estimating processing by using a modelcorresponding to the uncovered background area is similar to theprocessing indicated by the flowchart of FIG. 63 by using the normalequations corresponding to a model of the uncovered background area, andan explanation thereof is thus omitted.

[0557] The embodiment has been described, assuming that the objectcorresponding to the background is stationary. However, theabove-described mixture-ratio calculation processing can be applied evenif the image corresponding to the background area contains motion. Forexample, if the image corresponding to the background area is uniformlymoving, the estimated-mixture-ratio processor 401 shifts the overallimage in accordance with this motion, and performs processing in amanner similar to the case in which the object corresponding to thebackground is stationary. If the image corresponding to the backgroundarea contains locally different motions, the estimated-mixture-ratioprocessor 401 selects the pixels corresponding to the motions as thepixels belonging to the mixed area, and executes the above-describedprocessing.

[0558] As described above, the mixture-ratio calculator 102 is able tocalculate the mixture ratio α, which is a feature quantity correspondingto each pixel, based on the input image and the area informationsupplied from the area specifying unit 101.

[0559] By utilizing the mixture ratio α, it is possible to separate theforeground components and the background components contained in thepixel values while maintaining the information of motion blur containedin the image corresponding to the moving object.

[0560] By combining the images based on the mixture ratio α, it is alsopossible to generate an image which contains correct motion blur thatcoincides with the speed of a moving object and which faithfullyreflects the real world.

[0561] The foreground/background separator 105 is discussed below. FIG.64 is a block diagram illustrating an example of the configuration ofthe foreground/background separator 105. The input image supplied to theforeground/background separator 105 is supplied to a separating portion601, a switch 602, and a switch 604. The area information supplied fromthe area specifying unit 103 and indicating the information of thecovered background area and the uncovered background area is supplied tothe separating portion 601. The area information indicating theforeground area is supplied to the switch 602. The area informationindicating the background area supplied to the switch 604.

[0562] The mixture ratio α supplied from the mixture-ratio calculator104 is supplied to the separating portion 601.

[0563] The separating portion 601 separates the foreground componentsfrom the input image based on the area information indicating thecovered background area, the area information indicating the uncoveredbackground area, and the mixture ratio α, and supplies the separatedforeground components to a synthesizer 603. The separating portion 601also separates the background components from the input image, andsupplies the separated background components to a synthesizer 605.

[0564] The switch 602 is closed when a pixel corresponding to theforeground is input based on the area information indicating theforeground area, and supplies only the pixels corresponding to theforeground contained in the input image to the synthesizer 603.

[0565] The switch 604 is closed when a pixel corresponding to thebackground is input based on the area information indicating thebackground area, and supplies only the pixels corresponding to thebackground contained in the input image to the synthesizer 605.

[0566] The synthesizer 603 synthesizes a foreground component imagebased on the foreground components supplied from the separating portion601 and the pixels corresponding to the foreground supplied from theswitch 602, and outputs the synthesized foreground component image.Since the foreground area and the mixed area do not overlap, thesynthesizer 603 applies, for example, logical OR to the foregroundcomponents and the foreground pixels, thereby synthesizing theforeground component image.

[0567] In the initializing processing executed at the start of thesynthesizing processing for the foreground component image, thesynthesizer 603 stores an image whose pixel values are all 0 in abuilt-in frame memory. Then, in the synthesizing processing for theforeground component image, the synthesizer 603 stores the foregroundcomponent image (overwrites the previous image by the foregroundcomponent image). Accordingly, 0 is stored in the pixels correspondingto the background area in the foreground component image output from thesynthesizer 603.

[0568] The synthesizer 605 synthesizes a background component imagebased on the background components supplied from the separating portion601 and the pixels corresponding to the background supplied from theswitch 604, and outputs the synthesized background component image.Since the background area and the mixed area do not overlap, thesynthesizer 605 applies, for example, logical OR to the backgroundcomponents and the background pixels, thereby synthesizing thebackground component image.

[0569] In the initializing processing executed at the start of thesynthesizing processing for the background component image, thesynthesizer 605 stores an image whose pixel values are all 0 in abuilt-in frame memory. Then, in the synthesizing processing for thebackground component image, the synthesizer 605 stores the backgroundcomponent image (overwrites the previous image by the backgroundcomponent image). Accordingly, 0 is stored in the pixels correspondingto the foreground area in the background component image output from thesynthesizer 605.

[0570]FIG. 65A illustrates the input image input into theforeground/background separator 105 and the foreground component imageand the background component image output from the foreground/backgroundseparator 105. FIG. 65B illustrates a model corresponding to the inputimage input into the foreground/background separator 105 and theforeground component image and the background component image outputfrom the foreground/background separator 105.

[0571]FIG. 65A is a schematic diagram illustrating the image to bedisplayed, and FIG. 65B is a model obtained by expanding in the timedirection the pixels disposed in one line including the pixels belongingto the foreground area, the pixels belonging to the background area, andthe pixels belonging to the mixed area corresponding to FIG. 65A.

[0572] As shown in FIGS. 65A and 65B, the background component imageoutput from the foreground/background separator 105 consists of thepixels belonging to the background area and the background componentscontained in the pixels of the mixed area.

[0573] As shown in FIGS. 65A and 65B, the foreground component imageoutput from the foreground/background separator 105 consists of thepixel belonging to the foreground area and the foreground componentscontained in the pixels of the mixed area.

[0574] The pixel values of the pixels in the mixed area are separatedinto the background components and the foreground components by theforeground/background separator 105. The separated background componentsform the background component image together with the pixels belongingto the background area. The separated foreground components form theforeground component image together with the pixels belonging to theforeground area.

[0575] As discussed above, in the foreground component image, the pixelvalues of the pixels corresponding to the background area are set to 0,and significant pixel values are set in the pixels corresponding to theforeground area and the pixels corresponding to the mixed area.Similarly, in the background component image, the pixel values of thepixels corresponding to the foreground area are set to 0, andsignificant pixel values are set in the pixels corresponding to thebackground area and the pixels corresponding to the mixed area.

[0576] A description is given below of the processing executed by theseparating portion 601 for separating the foreground components and thebackground components from the pixels belonging to the mixed area.

[0577]FIG. 66 illustrates a model of an image indicating foregroundcomponents and background components in two frames including aforeground object moving from the left to the right in FIG. 66. In themodel of the image shown in FIG. 66, the amount of movement v is 4, andthe number of virtual divided portions is 4.

[0578] In frame #n, the leftmost pixel and the fourteenth througheighteenth pixels from the left consist of only the backgroundcomponents and belong to the background area. In frame #n, the secondthrough fourth pixels from the left contain the background componentsand the foreground components, and belong to the uncovered backgroundarea. In frame #n, the eleventh through thirteenth pixels from the leftcontain background components and foreground components, and belong tothe covered background area. In frame #n, the fifth through tenth pixelsfrom the left consist of only the foreground components, and belong tothe foreground area.

[0579] In frame #n+1, the first through fifth pixels from the left andthe eighteenth pixel from the left consist of only the backgroundcomponents, and belong to the background area. In frame #n+1, the sixththrough eighth pixels from the left contain background components andforeground components, and belong to the uncovered background area. Inframe #n+1, the fifteenth through seventeenth pixels from the leftcontain background components and foreground components, and belong tothe covered background area. In frame #n+1, the ninth through fourteenthpixels from the left consist of only the foreground components, andbelong to the foreground area.

[0580]FIG. 67 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the covered background area. InFIG. 67, α1 through α18 indicate mixture-ratios of the individual pixelsof frame #n. In FIG. 67, the fifteenth through seventeenth pixels fromthe left belong to the covered background area.

[0581] The pixel value C15 of the fifteenth pixel from the left in frame#n can be expressed by equation (68): $\begin{matrix}\begin{matrix}{{C15} = {{{B15}/v} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {B15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{{\alpha 15} \cdot {P15}} + {{F09}/v} + {{F08}/v} + {{F07}/v}}}\end{matrix} & (68)\end{matrix}$

[0582] where α15 indicates the mixture ratio of the fifteenth pixel fromthe left in frame #n, and P15 designates the pixel value of thefifteenth pixel from the left in frame #n−1.

[0583] The sum f15 of the foreground components of the fifteenth pixelfrom the left in frame #n can be expressed by equation (69) based onequation (68). $\begin{matrix}\begin{matrix}{{f15} = {{{F09}/v} + {{F08}/v} + {{F07}/v}}} \\{= {{C15} - {{\alpha 15} \cdot {P15}}}}\end{matrix} & (69)\end{matrix}$

[0584] Similarly, the sum f16 of the foreground components of thesixteenth pixel from the left in frame #n can be expressed by equation(70), and the sum f17 of the foreground components of the seventeenthpixel from the left in frame #n can be expressed by equation (71).

f16=C16−α16·P16  (70)

f17=C17−α17·P17  (71)

[0585] In this manner, the foreground components fc contained in thepixel value C of the pixel belonging to the covered background area canbe expressed by equation (72):

fc=C−α·P  (72)

[0586] where P designates the pixel value of the corresponding pixel inthe previous frame.

[0587]FIG. 68 illustrates the processing for separating the foregroundcomponents from the pixels belonging to the uncovered background area.In FIG. 68, α1 through α18 indicate mixture-ratios of the individualpixels of frame #n. In FIG. 68, the second through fourth pixels fromthe left belong to the uncovered background area.

[0588] The pixel value C02 of the second pixel from the left in frame #ncan be expressed by equation (73): $\begin{matrix}\begin{matrix}{{C02} = {{{B02}/v} + {{B02}/v} + {{B02}/v} + {{F01}/v}}} \\{= {{\alpha \quad {2 \cdot {B02}}} + {{F01}/v}}} \\{= {{\alpha \quad {2 \cdot {N02}}} + {{F01}/v}}}\end{matrix} & (73)\end{matrix}$

[0589] where α2 indicates the mixture ratio of the second pixel from theleft in frame #n, and N02 designates the pixel value of the second pixelfrom the left in frame #n+1.

[0590] The sum f02 of the foreground components of the second pixel fromthe left in frame #n can be expressed by equation (74) based on equation(73). $\begin{matrix}\begin{matrix}{{f02} = {{F01}/v}} \\{= {{C02} - {{\alpha 2} \cdot {N02}}}}\end{matrix} & (74)\end{matrix}$

[0591] Similarly, the sum f03 of the foreground components of the thirdpixel from the left in frame #n can be expressed by equation (75), andthe sum f04 of the foreground components of the fourth pixel from theleft in frame #n can be expressed by equation (76).

f03=C03−α3·N03  (75)

f04=C04−α4·N04  (76)

[0592] In this manner, the foreground components fu contained in thepixel value C of the pixel belonging to the uncovered background areacan be expressed by equation (77):

fu=C−α·N  (77)

[0593] where N designates the pixel value of the corresponding pixel inthe subsequent frame.

[0594] As discussed above, the separating portion 601 is able toseparate the foreground components from the pixels belonging to themixed area and the background components from the pixels belonging tothe mixed area based on the information indicating the coveredbackground area and the information indicating the uncovered backgroundarea contained in the area information, and the mixture ratio α for eachpixel.

[0595]FIG. 69 is a block diagram illustrating an example of theconfiguration of the separating portion 601 for executing theabove-described processing. An image input into the separating portion601 is supplied to a frame memory 621, and the area informationindicating the covered background area and the uncovered background areasupplied from the mixture-ratio calculator 104 and the mixture ratio αare supplied to a separation processing block 622.

[0596] The frame memory 621 stores the input images in units of frames.When a frame to be processed is frame #n, the frame memory 621 storesframe #n−1, which is the frame one frame before frame #n, frame #n, andframe #n+1, which is the frame one frame after frame #n.

[0597] The frame memory 621 supplies the corresponding pixels in frame#n−1, frame #n, and frame #n+1 to the separation processing block 622.

[0598] The separation processing block 622 applies the calculationsdiscussed with reference to FIGS. 67 and 68 to the pixel values of thecorresponding pixels in frame #n−1, frame #n, and frame #n+1 suppliedfrom the frame memory 621 based on the area information indicating thecovered background area and the uncovered background area and themixture ratio α so as to separate the foreground components and thebackground components from the pixels belonging to the mixed area inframe #n, and supplies them to a frame memory 623.

[0599] The separation processing block 622 is formed of an uncoveredarea processor 631, a covered area processor 632, a synthesizer 633, anda synthesizer 634.

[0600] A multiplier 641 of the uncovered area processor 631 multipliesthe pixel value of the pixel in frame #n+1 supplied from the framememory 621 by the mixture ratio α, and outputs the resulting pixel valueto a switch 642. The switch 642 is closed when the pixel in frame #n(corresponding to the pixel in frame #n+1) supplied from the framememory 621 belongs to the uncovered background area, and supplies thepixel value multiplied by the mixture ratio α supplied from themultiplier 641 to a calculator 643 and the synthesizer 634. The valueobtained by multiplying the pixel value of the pixel in frame #n+1 bythe mixture ratio α output from the switch 642 is equivalent to thebackground components of the pixel value of the corresponding pixel inframe #n.

[0601] The calculator 643 subtracts the background components suppliedfrom the switch 642 from the pixel value of the pixel in frame #nsupplied from the frame memory 621 so as to obtain the foregroundcomponents. The calculator 643 supplies the foreground components of thepixel in frame #n belonging to the uncovered background area to thesynthesizer 633.

[0602] A multiplier 651 of the covered area processor 632 multiplies thepixel value of the pixel in frame #n−1 supplied from the frame memory621 by the mixture ratio α, and outputs the resulting pixel value to aswitch 652. The switch 652 is closed when the pixel in frame #n(corresponding to the pixel in frame #n−1) supplied from the framememory 621 belongs to the covered background area, and supplies thepixel value multiplied by the mixture ratio α supplied from themultiplier 651 to a calculator 653 and the synthesizer 634. The valueobtained by multiplying the pixel value of the pixel in frame #n−1 bythe mixture ratio α output from the switch 652 is equivalent to thebackground components of the pixel value of the corresponding pixel inframe #n.

[0603] The calculator 653 subtracts the background components suppliedfrom the switch 652 from the pixel value of the pixel in frame #nsupplied from the frame memory 621 so as to obtain the foregroundcomponents. The calculator 653 supplies the foreground components of thepixel in frame #n belonging to the covered background area to thesynthesizer 633.

[0604] The synthesizer 633 combines the foreground components of thepixels belonging to the uncovered background area and supplied from thecalculator 643 with the foreground components of the pixels belonging tothe covered background area and supplied from the calculator 653, andsupplies the synthesized foreground components to the frame memory 623.

[0605] The synthesizer 634 combines the background components of thepixels belonging to the uncovered background area and supplied from theswitch 642 with the background components of the pixels belonging to thecovered background area and supplied from the switch 652, and suppliesthe synthesized background components to the frame memory 623.

[0606] The frame memory 623 stores the foreground components and thebackground components of the pixels in the mixed area of frame #nsupplied from the separation processing block 622.

[0607] The frame memory 623 outputs the stored foreground components ofthe pixels in the mixed area in frame #n and the stored backgroundcomponents of the pixels in the mixed area in frame #n.

[0608] By utilizing the mixture ratio α, which indicates the featurequantity, the foreground components and the background componentscontained in the pixel values can be completely separated.

[0609] The synthesizer 603 combines the foreground components of thepixels in the mixed area in frame #n output from the separating portion601 with the pixels belonging to the foreground area so as to generate aforeground component image. The synthesizer 605 combines the backgroundcomponents of the pixels in the mixed area in frame #n output from theseparating portion 601 with the pixels belonging to the background areaso as to generate a background component image.

[0610]FIG. 70A illustrates an example of the foreground component imagecorresponding to frame #n in FIG. 66. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thepixel values are set to 0.

[0611] The second and fourth pixels from the left belong to theuncovered background area before the foreground and the background areseparated. Accordingly, the background components are set to 0, and theforeground components are maintained. The eleventh through thirteenthpixels from the left belong to the covered background area before theforeground and the background are separated. Accordingly, the backgroundcomponents are set to 0, and the foreground components are maintained.The fifth through tenth pixels from the left consist of only theforeground components, which are thus maintained.

[0612]FIG. 70B illustrates an example of the background component imagecorresponding to frame #n in FIG. 66. The leftmost pixel and thefourteenth pixel from the left consist of only the background componentsbefore the foreground and the background are separated, and thus, thebackground components are maintained.

[0613] The second through fourth pixels from the left belong to theuncovered background area before the foreground and the background areseparated. Accordingly, the foreground components are set to 0, and thebackground components are maintained. The eleventh through thirteenthpixels from the left belong to the covered background area before theforeground and the background are separated. Accordingly, the foregroundcomponents are set to 0, and the background components are maintained.The fifth through tenth pixels from the left consist of only theforeground components, and thus, the pixel values are set to 0.

[0614] The processing for separating the foreground and the backgroundexecuted by the foreground/background separator 105 is described belowwith reference to the flowchart of FIG. 71. In step S601, the framememory 621 of the separating portion 601 obtains an input image, andstores frame #n for which the foreground and the background areseparated together with the previous frame #n−1 and the subsequent frame#n+1.

[0615] In step S602, the separation processing block 622 of theseparating portion 601 obtains area information supplied from themixture-ratio calculator 104. In step S603, the separation processingblock 622 of the separating portion 601 obtains the mixture ratio αsupplied from the mixture-ratio calculator 104.

[0616] In step S604, the uncovered area processor 631 extracts thebackground components from the pixel values of the pixels belonging tothe uncovered background area supplied from the frame memory 621 basedon the area information and the mixture ratio α.

[0617] In step S605, the uncovered area processor 631 extracts theforeground components from the pixel values of the pixels belonging tothe uncovered background area supplied from the frame memory 621 basedon the area information and the mixture ratio α.

[0618] In step S606, the covered area processor 632 extracts thebackground components from the pixel values of the pixels belonging tothe covered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

[0619] In step S607, the covered area processor 632 extracts theforeground components from the pixel values of the pixels belonging tothe covered background area supplied from the frame memory 621 based onthe area information and the mixture ratio α.

[0620] In step S608, the synthesizer 633 combines the foregroundcomponents of the pixels belonging to the uncovered background areaextracted in the processing of step S605 with the foreground componentsof the pixels belonging to the covered background area extracted in theprocessing of step S607. The synthesized foreground components aresupplied to the synthesizer 603. The synthesizer 603 further combinesthe pixels belonging to the foreground area supplied via the switch 602with the foreground components supplied from the separating portion 601so as to generate a foreground component image.

[0621] In step S609, the synthesizer 634 combines the backgroundcomponents of the pixels belonging to the uncovered background areaextracted in the processing of step S604 with the background componentsof the pixels belonging to the covered background area extracted in theprocessing of step S606. The synthesized background components aresupplied to the synthesizer 605. The synthesizer 605 further combinesthe pixels belonging to the background area supplied via the switch 604with the background components supplied from the separating portion 601so as to generate a background component image.

[0622] In step S610, the synthesizer 603 outputs the foregroundcomponent image. In step S611, the synthesizer 605 outputs thebackground component image. The processing is then completed.

[0623] As discussed above, the foreground/background separator 105 isable to separate the foreground components and the background componentsfrom the input image based on the area information and the mixture ratioα, and outputs the foreground component image consisting of only theforeground components and the background component image consisting ofonly the background components.

[0624] Adjustments of the amount of motion blur from a foregroundcomponent image are described below.

[0625]FIG. 72 is a block diagram illustrating an example of theconfiguration of the motion-blur adjusting unit 106. The motion vectorand the positional information thereof supplied from the motion detector102 and the area information supplied from the area specifying unit 103are supplied to a unit-of-processing determining portion 801 and amodel-forming portion 802. The area information supplied from theforeground/background separator 105 is supplied to the adder 804.

[0626] The unit-of-processing determining portion 801 supplies, togetherwith the motion vector, the unit of processing that is generated basedon the motion vector and the positional information thereof and the areainformation to the model-forming portion 802. The unit-of-processingdetermining portion 801 supplies the generated unit of processing to theadder 804.

[0627] As an example indicated by “A” is shown in FIG. 73, the unit ofprocessing generated by the unit-of-processing determining portion 801indicates consecutive pixels disposed in the moving direction startingfrom the pixel corresponding to the covered background area of theforeground component image until the pixel corresponding to theuncovered background area, or indicates consecutive pixels disposed inthe moving direction starting from the pixel corresponding to theuncovered background area until the pixel corresponding to the coveredbackground area. The unit of processing is formed of two pieces of datawhich indicate, for example, the upper left point (which is the positionof the leftmost or the topmost pixel in the image designated by the unitof processing) and the lower right point.

[0628] The model-forming portion 802 forms a model based on the motionvector and the input unit of processing. More specifically, for example,the model-forming portion 802 may store in advance a plurality of modelsin accordance with the number of pixels contained in the unit ofprocessing, the number of virtual divided portions of the pixel value inthe time direction, and the number of foreground components for eachpixel. The model-forming portion 902 then may select the model in whichthe correlation between the pixel values and the foreground componentsis designated, such as that in FIG. 74, based on the unit of processingand the number of virtual divided portions of the pixel value in thetime direction.

[0629] It is now assumed, for example, that the number of pixelscorresponding to the unit of processing is 12, and that the amount ofmovement v within the shutter time is 5. Then, the model-forming portion802 sets the number of virtual divided portions to 5, and selects amodel formed of eight types of foreground components so that theleftmost pixel contains one foreground component, the second pixel fromthe left contains two foreground components, the third pixel from theleft contains three foreground components, the fourth pixel from theleft contains four pixel components, the fifth pixel from the leftcontains five foreground components, the sixth pixel from the leftcontains five foreground components, the seventh pixel from the leftcontains five foreground components, the eighth pixel from the leftcontains five foreground components, the ninth pixel from the leftcontains four foreground components, the tenth pixel from the leftcontains three foreground components, the eleventh pixel from the leftcontains two foreground components, and the twelfth pixel from the leftcontains one foreground component.

[0630] Instead of selecting a model from the prestored models, themodel-forming portion 802 may generate a model based on the motionvector and the unit of processing when the motion vector and the unit ofprocessing are supplied.

[0631] The model-forming portion 802 supplies the selected model to anequation generator 803.

[0632] The equation generator 803 generates an equation based on themodel supplied from the model-forming portion 802.

[0633] A description is given below, with reference to the model of theforeground component image shown in FIG. 74, of equations generated bythe equation generator 803 when the number of foreground components is8, the number of pixels corresponding to the unit of processing is 12,and the amount of movement v is 5.

[0634] When the foreground components contained in the foregroundcomponent image corresponding to the shutter time/v are F01/v throughF08/v, the relationships between F01/v through F08/v and the pixelvalues C01 through C12 can be expressed by equations (78) through (89).

C01=F01/v  (78)

C02=F02/v+F01/v  (79)

C03=F03/v+F02/v+F01v  (80)

C04=F04/v+F03/v+F02/v+F01v  (81)

C05=F05/v+F04/v+F03/v+F02/v+F01v  (82)

C06=F06/v+F05/v+F04/v+F03/v+F02/v  (83)

C07=F07/v+F06/v+F05/v+F04/v+F03/v  (84)

C08=F08/v+F07/v+F06/v+F05/v+F04/v  (85)

C09=F08/v+F07/v+F06/v+F05/v  (86)

C10=F08/v+F07/v+F06/v  (87)

C11=F08/v+F07/v  (88)

C12=F08/v  (89)

[0635] The equation generator 803 generates an equation by modifying thegenerated equations. The equations generated by the equation generator803 are indicated by equations (90) though (101).

C01=1·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (90)

C02=1·F01/v+1·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (91)

C03=1·F01/v+1·F02/v+1·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (92)

C04=1·F01/v+1·F02/v+1·F03/v+1·F04/v+0·F05/v+0·F06/v+0·F07/v+0·F08/v  (93)

C05=1·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+0·F06/v+0·F07/v+0·F08/v  (94)

C06=0·F01/v+1·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+0·F07/v+0·F08/v  (95)

C07=0·F01/v+0·F02/v+1·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+0·F08/v  (96)

C08=0·F01/v+0·F02/v+0·F03/v+1·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (97)

C09=0·F01/v+0·F02/v+0·F03/v+0·F04/v+1·F05/v+1·F06/v+1·F07/v+1·F08/v  (98)

C10=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+1·F06/v+1·F07/v+1·F08/v  (99)

C11=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+1·F07/v+1·F08/v  (100)

C12=0·F01/v+0·F02/v+0·F03/v+0·F04/v+0·F05/v+0·F06/v+0·F07/v+1·F08/v  (101)

[0636] Equations (90) through (101) can be expressed by equation (102).$\begin{matrix}{{C\quad j} = {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad {i/v}}}} & (102)\end{matrix}$

[0637] In equation (102), j designates the position of the pixel. Inthis example, j has one of the values from 1 to 12. In equation (102), idesignates the position of the foreground value. In this example, i hasone of the values from 1 to 8. In equation (102), aij has the value 0 or1 according to the values of i and j.

[0638] Equation (102) can be expressed by equation (103) inconsideration of the error. $\begin{matrix}{{C\quad j} = {{\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad {i/v}}} + {e\quad j}}} & (103)\end{matrix}$

[0639] In equation (103), ej designates the error contained in thedesignated pixel Cj.

[0640] Equation (103) can be modified into equation (104).$\begin{matrix}{{e\quad j} = {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad {i/v}}}}} & (104)\end{matrix}$

[0641] In order to apply the method of least squares, the square sum Eof the error is defined as equation (105). $\begin{matrix}{E = {\sum\limits_{j = 01}^{12}{e\quad j^{2}}}} & (105)\end{matrix}$

[0642] In order to minimize the error, the partial differential valueusing the variable Fk with respect to the square sum E of the errorshould be 0. Fk is determined so that equation (106) is satisfied.$\begin{matrix}\begin{matrix}{\frac{\partial E}{{\partial F}\quad k} = {2 \cdot {\sum\limits_{j = 01}^{12}{e\quad {j \cdot \frac{{\partial e}\quad j}{{\partial F}\quad k}}}}}} \\{= {2 \cdot {\sum\limits_{j = 01}^{12}\left\{ {{\left( {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad {i/v}}}} \right) \cdot \left( {{- a}\quad k\quad {j/v}} \right)} = 0} \right.}}}\end{matrix} & (106)\end{matrix}$

[0643] In equation (106), since the amount of movement v is a fixedvalue, equation (107) can be deduced. $\begin{matrix}{{\sum\limits_{j = 01}^{12}{a\quad k\quad {j \cdot \left( {{C\quad j} - {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad {i/v}}}} \right)}}} = 0} & (107)\end{matrix}$

[0644] To expand equation (107) and transpose the terms, equation (108)can be obtained. $\begin{matrix}{{\sum\limits_{j = 01}^{12}\left( {a\quad k\quad {j \cdot {\sum\limits_{i = 01}^{08}{a\quad i\quad {j \cdot F}\quad i}}}} \right)} = {v{\sum\limits_{j = 01}^{12}{a\quad k\quad {j \cdot C}\quad j}}}} & (108)\end{matrix}$

[0645] Equation (108) is expanded into eight equations by substitutingthe individual integers from 1 to 8 into k in equation (108). Theobtained eight equations can be expressed by one matrix equation. Thisequation is referred to as a “normal equation”.

[0646] An example of the normal equation generated by the equationgenerator 803 based on the method of least squares is indicated byequation (109). $\begin{matrix}{{\begin{bmatrix}5 & 4 & 3 & 2 & 1 & 0 & 0 & 0 \\4 & 5 & 4 & 3 & 2 & 1 & 0 & 0 \\3 & 4 & 5 & 4 & 3 & 2 & 1 & 0 \\2 & 3 & 4 & 5 & 4 & 3 & 2 & 1 \\1 & 2 & 3 & 4 & 5 & 4 & 3 & 2 \\0 & 1 & 2 & 3 & 4 & 5 & 4 & 3 \\0 & 0 & 1 & 2 & 3 & 4 & 5 & 4 \\0 & 0 & 0 & 1 & 2 & 3 & 4 & 5\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05} \\{F06} \\{F07} \\{F08}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 08}^{12}{C\quad i}} \\{\sum\limits_{i = 07}^{11}{C\quad i}} \\{\sum\limits_{i = 06}^{10}{C\quad i}} \\{\sum\limits_{i = 05}^{09}{C\quad i}} \\{\sum\limits_{i = 04}^{08}{C\quad i}} \\{\sum\limits_{i = 03}^{07}{C\quad i}} \\{\sum\limits_{i = 02}^{06}{C\quad i}} \\{\sum\limits_{i = 01}^{05}{C\quad i}}\end{bmatrix}}} & (109)\end{matrix}$

[0647] When equation (109) is expressed by A·F=v·C, C, A, And v areknown, and F is unknown. A and v are known when The model is formed,while C becomes known when the pixel value is input in the additionprocessing.

[0648] By calculating the foreground components according to the normalequation based on the method of least squares, the error contained inthe pixel C can be distributed.

[0649] The equation generator 803 supplies the normal equation generatedas discussed above to the adder 804.

[0650] The adder 804 sets, based on the unit of processing supplied fromthe unit-of-processing determining portion 801, the pixel value Ccontained in the foreground component image in the matrix equationsupplied from the equation generator 803. The adder 804 supplies thematrix in which the pixel value C is set to a calculator 805.

[0651] The calculator 805 calculates the foreground component Fi/v fromwhich motion blur is eliminated by the processing based on a solution,such as a sweep-out method (Gauss-Jordan elimination), so as to obtainFi corresponding to i indicating one of the integers from 1 to 8, whichis the pixel value from which motion blur is eliminated. The calculator805 then outputs the foreground component image consisting of the pixelvalues Fi without motion blur, such as that in FIG. 75, to a motion-bluradder 806 and a selector 807.

[0652] In the foreground component image without motion blur shown inFIG. 75, the reason for setting F01 through F08 in C03 through C10,respectively, is not to change the position of the foreground componentimage with respect to the screen. However, F01 through F08 may be set inany desired positions.

[0653] The motion-blur adder 806 is able to adjust the amount of motionblur by adding the amount v′ by which motion blur is adjusted, which isdifferent from the amount of movement v, for example, the amount v′ bywhich motion blur is adjusted, which is one half the value of the amountof movement v, or the amount v′ by which motion blur is adjusted, whichis irrelevant to the amount of movement v. For example, as shown in FIG.76, the motion-blur adder 806 divides the foreground pixel value Fiwithout motion blur by the amount v′ by which motion blur is adjusted soas to obtain the foreground component Fi/v′. The motion-blur adder 806then calculates the sum of the foreground components Fi/v′, therebygenerating the pixel value in which the amount of motion blur isadjusted. For example, when the amount v′ by which motion blur isadjusted is 3, the pixel value C02 is set to (F01)/v′, the pixel valueC3 is set to (F01+F02)/v′, the pixel value C04 is set to(F01+F02+F03)/v′, and the pixel value C05 is set to (F02+F03+F04)/v′.

[0654] The motion-blur adder 806 supplies the foreground component imagein which the amount of motion blur is adjusted to a selector 807.

[0655] The selector 807 selects one of the foreground component imagewithout motion blur supplied from the calculator 805 and the foregroundcomponent image in which the amount of motion blur is adjusted suppliedfrom the motion-blur adder 806 based on a selection signal reflecting auser's selection, and outputs the selected foreground component image.

[0656] As discussed above, the motion-blur adjusting unit 106 is able toadjust the amount of motion blur based on the selection signal and theamount v′ by which motion blur is adjusted.

[0657] Also, for example, when the number of pixels corresponding to theunit of processing is 8, and the amount of movement v is 4, as shown inFIG. 77, the motion-blur adjusting unit 106 generates a matrix equationexpressed by equation (110). $\begin{matrix}{{\begin{bmatrix}4 & 3 & 2 & 1 & 0 \\3 & 4 & 3 & 2 & 1 \\2 & 3 & 4 & 3 & 2 \\1 & 2 & 3 & 4 & 3 \\0 & 1 & 2 & 3 & 4\end{bmatrix}\begin{bmatrix}{F01} \\{F02} \\{F03} \\{F04} \\{F05}\end{bmatrix}} = {v \cdot \begin{bmatrix}{\sum\limits_{i = 05}^{08}{C\quad i}} \\{\sum\limits_{i = 04}^{07}{C\quad i}} \\{\sum\limits_{i = 03}^{06}{C\quad i}} \\{\sum\limits_{i = 02}^{05}{C\quad i}} \\{\sum\limits_{i = 01}^{04}{C\quad i}}\end{bmatrix}}} & (110)\end{matrix}$

[0658] In this manner, the motion-blur adjusting unit 106 calculates Fi,which is the pixel value in which the amount of motion blur is adjusted,by setting up the equation in accordance with the length of the unit ofprocessing. Similarly, for example, when the number of pixels containedin the unit of processing is 100, the equation corresponding to 100pixels is generated so as to calculate Fi.

[0659]FIG. 78 illustrates an example of another configuration of themotion-blur adjusting unit 106. The same elements as those shown in FIG.72 are designated with like reference numerals, and an explanationthereof is thus omitted.

[0660] Based on a selection signal, a selector 821 directly supplies aninput motion vector and a positional signal thereof to theunit-of-processing determining portion 801 and the model-forming portion802. Alternatively, the selector 821 may substitute the magnitude of themotion vector by the amount v′ by which motion blur is adjusted, andthen supplies the motion vector and the positional signal thereof to theunit-of-processing determining portion 801 and the model-forming unit802.

[0661] With this arrangement, the unit-of-processing determining portion801 through the calculator 805 of the motion-blur adjusting unit 106shown in FIG. 78 are able to adjust the amount of motion blur inaccordance with the amount of movement v and the amount v′ by whichmotion blur is adjusted. For example, when the amount of movement is 5,and the amount v′ by which motion blur is adjusted is 3, theunit-of-processing determining portion 801 through the calculator 805 ofthe motion-blur adjusting unit 106 shown in FIG. 76 execute computationon the foreground component image in which the amount of movement v is 5shown in FIG. 74 according to the model shown in FIG. 76 in which theamount v′ by which motion blur is adjusted is 3. As a result, the imagecontaining motion blur having the amount of movement v of (amount ofmovement v)/(amount v′ by which motion blur is adjusted)=5/3, i.e.,about 1.7 is obtained. In this case, the calculated image does notcontain motion blur corresponding to the amount of movement v of 3.Accordingly, it should be noted that the relationship between the amountof movement v and the amount v′ by which motion blur is adjusted isdifferent from the result of the motion-blur adder 806.

[0662] As discussed above, the motion-blur adjusting unit 106 generatesthe equation in accordance with the amount of movement v and the unit ofprocessing, and sets the pixel values of the foreground component imagein the generated equation, thereby calculating the foreground componentimage in which the amount of motion blur is adjusted.

[0663] The processing for adjusting the amount of motion blur containedin the foreground component image executed by the motion-blur adjustingunit 106 is described below with reference to the flowchart of FIG. 79.

[0664] In step S801, the unit-of-processing determining portion 801 ofthe motion-blur adjusting unit 106 generates the unit of processingbased on the motion vector and the area information, and supplies thegenerated unit of processing to the model-forming portion 802.

[0665] In step S802, the model-forming portion 802 of the motion-bluradjusting unit 106 selects or generates the model in accordance with theamount of movement v and the unit of processing. In step S803, theequation generator 803 generates the normal equation based on theselected model.

[0666] In step S804, the adder 804 sets the pixel values of theforeground component image in the generated normal equation. In stepS805, the adder 804 determines whether the pixel values of all thepixels corresponding to the unit of processing are set. If it isdetermined that the pixel values of all the pixels corresponding to theunit of processing are not yet set, the process returns to step S804,and the processing for setting the pixel values in the normal equationis repeated.

[0667] If it is determined in step S805 that the pixel values of all thepixels corresponding to the unit of processing are set, the processproceeds to step S806. In step S806, the calculator 805 calculates thepixel values of the foreground in which the amount of motion blur isadjusted based on the normal equation in which the pixel values are setsupplied from the adder 804. The processing is then completed.

[0668] As discussed above, the motion-blur adjusting unit 106 is able toadjust the amount of motion blur of the foreground image containingmotion blur based on the motion vector and the area information.

[0669] That is, it is possible to adjust the amount of motion blurcontained in the pixel values, that is, contained in sampled data.

[0670] As is seen from the foregoing description, the image processingapparatus shown in FIG. 2 is able to adjust the amount of motion blurcontained in the input image. The image processing apparatus configuredas shown in FIG. 2 is able to calculate the mixture ratio α, which isembedded information, and outputs the calculated mixture-ratio α.

[0671]FIG. 80 is a block diagram illustrating another example of theconfiguration of the motion-blur adjusting unit 106. The motion vectorand the positional information thereof supplied from the motion detector102 are supplied to a unit-of-processing determining portion 901 and anadjusting portion 905. The area information supplied from the areaspecifying unit 103 is supplied to the unit-of-processing determiningportion 901. The foreground component image supplied from theforeground/background separator 105 is supplied to a calculator 904.

[0672] The unit-of-processing determining portion 901 supplies, togetherwith the motion vector, the unit of processing generated based on themotion vector and the positional information thereof and the areainformation to a model-forming portion 902.

[0673] The model-forming portion 902 forms a model based on the motionvector and the input unit of processing. More specifically, for example,the model-forming portion 902 may store in advance a plurality of modelsin accordance with the number of pixels contained in the unit ofprocessing, the number of virtual divided portions of the pixel value inthe time direction, and the number of foreground components for eachpixel. The model-forming portion 902 then may select the model in whichthe correlation between the pixel values and the foreground componentsis designated, such as that in FIG. 81, based on the unit of processingand the number of virtual divided portions of the pixel value in thetime direction.

[0674] It is now assumed, for example, that the number of pixelscorresponding to the unit of processing is 12, and that the amount ofmovement v within the shutter time is 5. Then, the model-forming portion902 sets the number of virtual divided portions to 5, and selects amodel formed of eight types of foreground components so that theleftmost pixel contains one foreground component, the second pixel fromthe left contains two foreground components, the third pixel from theleft contains three foreground components, the fourth pixel from theleft contains four pixel components, the fifth pixel from the leftcontains five foreground components, the sixth pixel from the leftcontains five foreground components, the seventh pixel from the leftcontains five foreground components, the eighth pixel from the leftcontains five foreground components, the ninth pixel from the leftcontains four foreground components, the tenth pixel from the leftcontains three foreground components, the eleventh pixel from the leftcontains two foreground components, and the twelfth pixel from the leftcontains one foreground component.

[0675] Instead of selecting a model from the prestored models, themodel-forming portion 902 may generate a model based on the motionvector and the unit of processing when the motion vector and the unit ofprocessing are supplied.

[0676] An equation generator 903 generates an equation based on themodel supplied from the model-forming portion 902.

[0677] A description is now given, with reference to the models offoreground component images shown in FIGS. 81 through 83, of an exampleof the equation generated by the equation generator 903 when the numberof foreground components is 8, the number of pixels corresponding to theunit of processing is 12, and the amount of movement v is 5.

[0678] When the foreground components contained in the foregroundcomponent image corresponding to the shutter time/v are F01/v throughF08/v, the relationships between F01/v through F08/v and pixel valuesC01 through C12 can be expressed by equations (78) through (89), asstated above.

[0679] By considering the pixel values C12 and C11, the pixel value C12contains only the foreground component F08/v, as expressed by equation(111), and the pixel value C11 consists of the product sum of theforeground component F08/v and the foreground component F07/v.Accordingly, the foreground component F07/v can be found by equation(112).

F08/v=C12  (111)

F07/v=C11−C12  (112)

[0680] Similarly, by considering the foreground components contained inthe pixel values C10 through C01, the foreground components F06/vthrough F01/v can be found by equations (113) through (118),respectively.

F06/v=C10−C11  (113)

F05/v=C09−C10  (114)

F04/v=C08−C09  (115)

F03/v=C07−C08+C12  (116)

F02/v=C06−C07+C11−C12  (117)

F01/v=C05−C06+C10−C11  (118)

[0681] The equation generator 903 generates the equations forcalculating the foreground components by the difference between thepixel values, as indicated by the examples of equations (111) through(118). The equation generator-903 supplies the generated equations tothe calculator 904.

[0682] The calculator 904 sets the pixel values of the foregroundcomponent image in the equations supplied from the equation generator903 so as to obtain the foreground components based on the equations inwhich the pixel values are set. For example, when equations (111)through (118) are supplied from the equation generator 903, thecalculator 904 sets the pixel values C05 through C12 in equations (111)through (118).

[0683] The calculator 904 calculates the foreground components based onthe equations in which the pixel values are set. For example, thecalculator 904 calculates the foreground components F01/v through F08/v,as shown in FIG. 82, based on the calculations of equations (111)through (118) in which the pixel values C05 through C12 are set. Thecalculator 904 supplies the foreground components F01/v through F08/v tothe adjusting portion 905.

[0684] The adjusting portion 905 multiplies the foreground componentssupplied from the calculator 904 by the amount of movement v containedin the motion vector supplied from the unit-of-processing determiningportion 901 so as to obtain the foreground pixel values from whichmotion blur is eliminated. For example, when the foreground componentsF01/v through F08/v are supplied from the calculator 904, the adjustingportion 905 multiples each of the foreground components F01/v throughF08/v by the amount of movement v, i.e., 5, so as to obtain theforeground pixel values F01 through F08 from which motion blur iseliminated, as shown in FIG. 83.

[0685] The adjusting portion 905 supplies the foreground component imageconsisting of the foreground pixel values without motion blur calculatedas described above to a motion-blur adder 906 and a selector 907.

[0686] The motion-blur adder 906 is able to adjust the amount of motionblur by using the amount v′ by which motion blur is adjusted, which isdifferent from the amount of movement v, for example, the amount v′ bywhich motion blur is adjusted, which is one half the value of the amountof movement v, or the amount v′ by which motion blur is adjusted, whichis irrelevant to the amount of movement v. For example, as shown in FIG.76, the motion-blur adder 906 divides the foreground pixel value Fiwithout motion blur by the amount v′ by which motion blur is adjusted soas to obtain the foreground component Fi/v′. The motion-blur adder 906then calculates the sum of the foreground components Fi/v′, therebygenerating the pixel value in which the amount of motion blur isadjusted. For example, when the amount v′ by which motion blur isadjusted is 3, the pixel value C02 is set to (F01)/v′, the pixel valueC3 is set to (F01+F02)/v′, the pixel value C04 is set to(F01+F02+F03)/v′, and the pixel value C05 is set to (F02+F03+F04)/v′.

[0687] The motion-blur adder 906 supplies the foreground component imagein which the amount of motion blur is adjusted to the selector 907.

[0688] The selector 907 selects either the foreground component imagewithout motion blur supplied from the adjusting portion 905 or theforeground component image in which the amount of motion blur isadjusted supplied from the motion-blur adder 906 based on a selectionsignal reflecting a user's selection, and outputs the selectedforeground component image.

[0689] As discussed above, the motion-blur adjusting unit 106 is able toadjust the amount of motion blur based on the selection signal and theamount v′ by which motion blur is adjusted.

[0690] The processing for adjusting the amount of motion blur of theforeground executed by the motion-blur adjusting unit 106 configured asshown in FIG. 80 is described below with reference to the flowchart ofFIG. 84.

[0691] In step S901, the unit-of-processing determining portion 901 ofthe motion-blur adjusting unit 106 generates the unit of processingbased on the motion vector and the area information, and supplies thegenerated unit of processing to the model-forming portion 902 and theadjusting portion 905.

[0692] In step S902, the model-forming portion 902 of the motion-bluradjusting unit 106 selects or generates the model according to theamount of movement v and the unit of processing. In step S903, theequation generator 903 generates, based on the selected or generatedmodel, the equations for calculating the foreground components by thedifference between the pixel values of the foreground component image.

[0693] In step S904, the calculator 904 sets the pixel values of theforeground component image in the generated equations, and extracts theforeground components by using the difference between the pixel valuesbased on the equations in which the pixel values are set. In step S905,the calculator 904 determines whether all the foreground componentscorresponding to the unit of processing have been extracted. If it isdetermined that all the foreground components corresponding to the unitof processing have not been extracted, the process returns to step S904,and the processing for extracting the foreground components is repeated.

[0694] If it is determined in step S905 that all the foregroundcomponents corresponding to the unit of processing have been extracted,the process proceeds to step S906. In step S906, the adjusting portion905 adjusts each of the foreground components F01/v through F08/vsupplied from the calculator 904 based on the amount of movement v so asto obtain the foreground pixel values F01/v through F08/v from whichmotion blur is eliminated.

[0695] In step S907, the motion-blur adder 906 calculates the foregroundpixel values in which the amount of motion blur is adjusted, and theselector 907 selects the image without motion blur or the image in whichthe amount of motion blur is adjusted, and outputs the selected image.The processing is then completed.

[0696] As described above, the motion-blur adjusting unit 106 configuredas shown in FIG. 80 is able to more speedily adjust motion blur of theforeground image containing motion blur according to simplercomputations.

[0697] A known technique for partially eliminating motion blur, such asa Wiener filter, is effective when being used in the ideal state, but isnot sufficient for an actual image quantized and containing noise. Incontrast, it is proved that the motion-blur adjusting unit 106configured as shown in FIG. 80 is sufficiently effective for an actualimage quantized and containing noise. It is thus possible to eliminatemotion blur with high precision.

[0698]FIG. 85 is a block diagram illustrating another configuration ofthe function of the image processing apparatus.

[0699] The elements similar to those shown in FIG. 2 are designated withlike reference numerals, and an explanation thereof is thus omitted.

[0700] The area specifying unit 103 supplies area information to themixture-ratio calculator 104 and a synthesizer 1001.

[0701] The mixture-ratio calculator 104 supplies the mixture ratio α tothe foreground/background separator 105 and the synthesizer 1001.

[0702] The foreground/background separator 105 supplies the foregroundcomponent image to the synthesizer 1001.

[0703] The synthesizer 1001 combines a certain background image with theforeground component image supplied from the foreground/backgroundseparator 105 based on the mixture ratio α supplied from themixture-ratio calculator 104 and the area information supplied from thearea specifying unit 103, and outputs the synthesized image in which thecertain background image and the foreground component image arecombined.

[0704]FIG. 86 illustrates the configuration of the synthesizer 1001. Abackground component generator 1021 generates a background componentimage based on the mixture ratio α and a certain background image, andsupplies the background component image to a mixed-area-imagesynthesizing portion 1022.

[0705] The mixed-area-image synthesizing portion 1022 combines thebackground component image supplied from the background componentgenerator 1021 with the foreground component image so as to generate amixed-area synthesized image, and supplies the generated mixture-areasynthesized image to an image synthesizing portion 1023.

[0706] The image synthesizer 1023 combines the foreground componentimage, the mixed-area synthesized image supplied from themixed-area-image synthesizing portion 1022, and the certain backgroundimage based on the area information so as to generate a synthesizedimage, and outputs it.

[0707] As discussed above, the synthesizer 1001 is able to combine theforeground component image with a certain background image.

[0708] The image obtained by combining a foreground component image witha certain background image based on the mixture ratio α, which is thefeature quantity, appears more natural compared to an image obtained bysimply combining pixels.

[0709]FIG. 87 is a block diagram illustrating still anotherconfiguration of the function of the image processing apparatus foradjusting the amount of motion blur. The image processing apparatusshown in FIG. 2 sequentially performs the area-specifying operation andthe calculation for the mixture ratio α. In contrast, the imageprocessing apparatus shown in FIG. 87 simultaneously performs thearea-specifying operation and the calculation for the mixture ratio α.

[0710] The functional elements similar to those in the block diagram ofFIG. 2 are designated with like reference numerals, and an explanationthereof is thus omitted.

[0711] An input image is supplied to a mixture-ratio calculator 1101, aforeground/background separator 1102, the area specifying unit 103, andthe object extracting unit 101.

[0712] The mixture-ratio calculator 1101 calculates, based on the inputimage, the estimated mixture ratio when it is assumed that each pixelcontained in the input image belongs to the covered background area, andthe estimated mixture ratio when it is assumed that each pixel containedin the input image belongs to the uncovered background area, andsupplies the estimated mixture ratios calculated as described above tothe foreground/background separator 1102.

[0713]FIG. 88 is a block diagram illustrating the configuration of themixture-ratio calculator 1101.

[0714] An estimated-mixture-ratio processor 401 shown in FIG. 88 is thesame as the estimated-mixture-ratio processor 401 shown in FIG. 47. Anestimated-mixture-ratio processor 402 shown in FIG. 88 is the same asthe estimated-mixture-ratio processor 402 shown in FIG. 47.

[0715] The estimated-mixture-ratio processor 401 calculates theestimated mixture ratio for each pixel by the computation correspondingto a model of the covered background area based on the input image, andoutputs the calculated estimated mixture ratio.

[0716] The estimated-mixture-ratio processor 402 calculates theestimated mixture ratio for each pixel by the computation correspondingto a model of the uncovered background area based on the input image,and outputs the calculated estimated mixture ratio.

[0717] The foreground/background separator 1102 generates the foregroundcomponent image from the input image based on the estimated mixtureratio calculated when it is assumed that the pixel belongs to thecovered background area supplied from the mixture-ratio calculator 1101,the estimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and the area information supplied from the areaspecifying unit 103, and supplies the generated foreground componentimage to the motion-blur adjusting unit 106 and the selector 107.

[0718]FIG. 89 is a block diagram illustrating an example of theconfiguration of the foreground/background separator 1102.

[0719] The elements similar to those of the foreground/backgroundseparator 105 shown in FIG. 64 are designated with like referencenumerals, and an explanation thereof is thus omitted.

[0720] A selector 1121 selects, based on the area information suppliedfrom the area specifying unit 103, either the estimated mixture ratiocalculated when it is assumed that the pixel belongs to the coveredbackground area supplied from the mixture-ratio calculator 1101 or theestimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and supplies the selected estimated mixture ratio tothe separating portion 601 as the mixture ratio α.

[0721] The separating portion 601 extracts the foreground components andthe background components from the pixel values of the pixels belongingto the mixed area based on the mixture ratio α supplied from theselector 1121 and the area information, and supplies the extractedforeground components to the synthesizer 603 and also supplies theforeground components to the synthesizer 605.

[0722] The separating portion 601 can be configured similarly to thecounterpart shown in FIG. 69.

[0723] The synthesizer 603 synthesizes the foreground component imageand outputs it. The synthesizer 605 synthesizes the background componentimage and outputs it.

[0724] The motion-blur adjusting unit 106 shown in FIG. 87 can beconfigured similarly to the counterpart shown in FIG. 2. The motion-bluradjusting unit 106 adjusts the amount of motion blur contained in theforeground component image supplied from the foreground/backgroundseparator 1102 based on the area information and the motion vector, andoutputs the foreground component image in which the amount of motionblur is adjusted.

[0725] The selector 107 shown in FIG. 87 selects the foregroundcomponent image supplied from the foreground/background separator 1102or the foreground component image in which the amount of motion blur isadjusted supplied from the motion-blur adjusting unit 106 based on, forexample, a selection signal reflecting a user's selection, and outputsthe selected foreground component image.

[0726] As discussed above, the image processing apparatus shown in FIG.87 is able to adjust the amount of motion blur contained in an imagecorresponding to a foreground object of the input image, and outputs theresulting foreground object image. The image processing apparatus havingthe configuration shown in FIG. 87 is able to calculate the mixtureratio α, which is embedded information, and outputs the calculatedmixture-ratio α in the same manner as in the first embodiment.

[0727]FIG. 90 is a block diagram illustrating still anotherconfiguration of the function of the image processing apparatus foradjusting the amount of motion blur. The image processing apparatusshown in FIG. 85 sequentially performs the area-specifying operation andthe calculation for the mixture ratio α. In contrast, the imageprocessing apparatus shown in FIG. 90 simultaneously performs thearea-specifying operation and the calculation for the mixture ratio α.

[0728] The functional elements similar to those indicated by the blockof FIG. 87 are designated with like reference numerals, and anexplanation thereof is thus omitted.

[0729] The mixture-ratio calculator 1101 shown in FIG. 90 calculates,based on the input image, the estimated mixture ratio when it is assumedthat each pixel contained in the input image belongs to the coveredbackground area, and the estimated mixture ratio when it is assumed thateach pixel contained in the input image belongs to the uncoveredbackground area, and supplies the estimated mixture ratios calculated asdescribed above to the foreground/background separator 1102 and asynthesizer 1201.

[0730] The foreground/background separator 1102 shown in FIG. 90generates the foreground component image from the input image based onthe estimated mixture ratio calculated when it is assumed that the pixelbelongs to the covered background area supplied from the mixture-ratiocalculator 1101, the estimated mixture ratio calculated when it isassumed that the pixel belongs to the uncovered background area suppliedfrom the mixture-ratio calculator 1101, and the area informationsupplied from the area specifying unit 103, and supplies the generatedforeground component image to the synthesizer 1201.

[0731] The synthesizer 1201 combines a certain background image with theforeground component image supplied from the foreground/backgroundseparator 1102 based on the estimated mixture ratio calculated when itis assumed that the pixel belongs to the covered background areasupplied from the mixture-ratio calculator 1101, the estimated mixtureratio calculated when it is assumed that the pixel belongs to theuncovered background area supplied from the mixture-ratio calculator1101, and the area information supplied from the area specifying unit103, and outputs the synthesized image in which the background image andthe foreground component image are combined.

[0732]FIG. 91 illustrates the configuration of the synthesizer 1201. Thefunctional elements similar to those of the block diagram of FIG. 86 aredesignated with like reference numerals, and explanation thereof is thusomitted.

[0733] A selector 1221 selects, based on the area information suppliedfrom the area specifying unit 103, either the estimated mixture ratiocalculated when it is assumed that the pixel belongs to the coveredbackground area supplied from the mixture-ratio calculator 1101 or theestimated mixture ratio calculated when it is assumed that the pixelbelongs to the uncovered background area supplied from the mixture-ratiocalculator 1101, and supplies the selected estimated mixture ratio tothe background component generator 1021 as the mixture ratio α.

[0734] The background component generator 1021 shown in FIG. 91generates a background component image based on the mixture ratio αsupplied from the selector 1221 and a certain background image, andsupplies the background component image to the mixed-area-imagesynthesizing portion 1022.

[0735] The mixed-area-image synthesizing portion 1022 shown in FIG. 91combines the background component image supplied from the backgroundcomponent generator 1021 with the foreground component image so as togenerate a mixed-area synthesized image, and supplies the generatedmixed-area synthesized image to the image synthesizing portion 1023.

[0736] The image synthesizing portion 1023 combines the foregroundcomponent image, the mixed-area synthesized image supplied from themixed-area-image synthesizing portion 1022, and the background imagebased on the area information so as to generate a synthesized image andoutputs it.

[0737] In this manner, the synthesizer 1201 is able to combine theforeground component image with a certain background image.

[0738] A description is given of an image processing apparatus forprocessing an input image, which is input as component signals,according to the present invention.

[0739] In this specification, the component refers to an individualsignal, such as a luminance signal, a color-difference signal, or an RGBsignal in the component signal.

[0740] A description is given below based on an example in whichcomponent 1 is a luminance value Y, component 2 is a color difference U,and component 3 is a color difference V.

[0741]FIG. 92 shows an embodiment of an image processing apparatus forgenerating area information on the basis of an input image which isinput as component signals.

[0742] The component 1, which is one of the component signals of theinput image, is input to an area specifying unit 103-1. The component 2,which is another one of the component signals of the input image, isinput to an area specifying unit 103-2. The component 3, which is stillanother one of the component signals of the input image, is input to anarea specifying unit 103-3.

[0743] Based on the component 1, the area specifying unit 103-1generates area information 1 and supplies the generated area information1 to a logical OR processor 1301. The area specifying unit 103-1 has thesame configuration as that of the area specifying unit 103, and anexplanation thereof is thus omitted.

[0744] Based on the component 2, the area specifying unit 103-2generates area information 2 and supplies the generated area information2 to a logical OR processor 1302. The area specifying unit 103-2 has thesame configuration as that of the area specifying unit 103, and anexplanation thereof is thus omitted.

[0745] Based on the component 3, the area specifying unit 103-3generates area information 3 and supplies the generated area information3 to a logical OR processor 1303. The area specifying unit 103-3 has thesame configuration as that of the area specifying unit 103, and anexplanation thereof is thus omitted.

[0746] Based on the area information 1 supplied from the area specifyingunit 103-1, the area information 2 supplied from the area specifyingunit 103-2, and the area information 3 supplied from the area specifyingunit 103-3, the logical OR processor 1301 calculates the logical OR ofthe foreground area indicated by the area information 1, the foregroundarea indicated by the area information 2, and the foreground areaindicated by the area information 3, and generates area information inwhich the foreground area calculated by logical OR is set. Based on thearea information 1 supplied from the area specifying unit 103-1, thearea information 2 supplied from the area specifying unit 103-2, and thearea information 3 supplied from the area specifying unit 103-3, thelogical OR processor 1301 computes the logical OR of the background areaindicated by the area information 1, the background area indicated bythe area information 2, and the background area indicated by the areainformation 3, and generates area information in which the backgroundarea calculated by logical OR is set.

[0747] Based on the area information 1 supplied from the area specifyingunit 103-1, the area information 2 supplied from the area specifyingunit 103-2, and the area information 3 supplied from the area specifyingunit 103-3, the logical OR processor 1301 computes the logical OR of thecovered background area indicated by the area information 1, the coveredbackground area indicated by the area information 2, and the coveredbackground area indicated by the area information 3, and generates areainformation in which the covered background area calculated by logicalOR is set. Based on the area information 1 supplied from the areaspecifying unit 103-1, the area information 2 supplied from the areaspecifying unit 103-2, and the area information 3 supplied from the areaspecifying unit 103-3, the logical OR processor 1301 computes thelogical OR of the uncovered background area indicated by the areainformation 1, the uncovered background area indicated by the areainformation 2, and the uncovered background area indicated by the areainformation 3, and generates area information in which the uncoveredbackground area calculated by logical OR is set.

[0748] The logical OR processor 1301 outputs the area information inwhich the foreground area, the background area, the covered backgroundarea, and the uncovered background area are set.

[0749]FIG. 93 illustrates the relationship among the component 1, thecomponent 2, and the component 3, contained in the component signals. InFIG. 93, reference letter A denotes the component 1, reference letter Bdenotes the component 2, and reference letter C denotes the component 3.

[0750] A sensor has, for example, three CCD area sensors correspondingto the component 1, the component 2, and the component 3. Thecharacteristics of the CCD area sensor corresponding to the component 1,the characteristics of the CCD area sensor corresponding to thecomponent 2, and the characteristics of the CCD area sensorcorresponding to the component 3 are the same, and distortions whichoccur in the component 1, the component 2, and the component 3 are thesame.

[0751] The sensor captures the image of an object 111 corresponding to asingle foreground and an object 112 corresponding to a singlebackground, and outputs the component 1, the component 2, and thecomponent 3.

[0752] The real world including the object 111 corresponding to a singleforeground and the object 112 corresponding to a single background isone, and the phenomenon which occurs in the real world is one. That is,for example, the shape of the object 111 corresponding to a singleforeground is one, and the motion of the object 111 corresponding to aforeground is one.

[0753] Therefore, when the image of the object 111 corresponding to asingle foreground and the image of the object 112 corresponding to asingle background are captured by the sensor, the foreground area, thebackground area, the mixed area, and the mixture ratio α of thecomponent 1, the foreground area, the background area, the mixed area,and the mixture ratio α of the component 2, and the foreground area, thebackground area, the mixed area, and the mixture ratio α of thecomponent 3 are the same as each other.

[0754] However, when the same processing is performed based on each ofthe component 1, the component 2, and the component 3, the same areainformation and the same mixture ratio α cannot always be calculated foreach.

[0755] For example, when the image of the object 111 corresponding to aforeground and the image of the object 112 corresponding to abackground, having chroma in which the color-difference signalcorresponding to the component 3, which is a color-difference signal, isapproximately 0 are captured, the component 1 which is a luminancesignal and the component 2 which is a color-difference signal containsignificant values, but the component 3 which is a color-differencesignal scarcely contains significant values. In such a case, even if thearea signal and the mixture ratio α are calculated based on thecomponent 3, significant values cannot be determined.

[0756] Furthermore, for example, when the component 1 which is aluminance signal scarcely changes in the time direction and in thespatial direction and only the color-difference signal changes in thetime direction or in the spatial direction, even if the area signal andthe mixture ratio α are calculated based on the component 1, significantvalues cannot be determined, and if the area signal and the mixtureratio α are calculated based on the component 2 or 3, significant valuescan be determined.

[0757] Furthermore, there are cases in which the CCD area sensorcorresponding to a specific component may cause an error to occur.

[0758] As discussed above, as a result of performing processing using aplurality of components corresponding to single area information and asingle mixture ratio α to be calculated, results with a higher accuracycompared to processing using, for example, a single component signal ora single composite signal are obtained.

[0759] When a statistical process is performed using a plurality ofcomponents, since the amount of data is increased, the accuracy of theprocessed results is improved further. Furthermore, for example, theaccuracy of the mixture ratio α which is calculated by applying a methodof least squares to a plurality of components is higher than theaccuracy of the mixture ratio α calculated by processing using a singlecomponent signal or a single composite signal.

[0760] Next, referring to the flowchart in FIG. 94, a description willnow be given of area determining processing using a component signal bythe image processing apparatus configured as shown in FIG. 92.

[0761] In step S1301, the area specifying unit 103-1 executes areaspecifying processing on the basis of the component 1 in order togenerate area information 1, and supplies the generated area information1 to the logical OR processor 1301. The processing of step S1301 is thesame as the processing of step S11, and a detailed explanation thereofis thus omitted.

[0762] In step S1302, the area specifying unit 103-2 executes areaspecifying processing based on the component 2 in order to generate areainformation 2, and supplies the generated area information 2 to thelogical OR processor 1301. The processing of step S1302 is the same asthe processing of step S11, and a detailed explanation thereof is thusomitted.

[0763] In step S1303, the area specifying unit 103-3 executes areaspecifying processing based on the component 3 in order to generate areainformation 3, and supplies the generated area information 3 to thelogical OR processor 1301. The processing of step S1303 is the same asthe processing of step S11, and an explanation thereof is thus omitted.

[0764] In step S1304, the logical OR processor 1301 calculates thelogical OR of the foreground area specified by the component 1, theforeground area specified by the component 2, and the foreground areaspecified by the component 3, and sets the foreground area calculated bylogical OR as area information.

[0765] In step S1305, the logical OR processor 1301 calculates thelogical OR of the background area specified by the component 1, thebackground area specified by the component 2, and the background areaspecified by the component 3, and sets the background area calculated bylogical OR as area information.

[0766] In step S1306, the logical OR processor 1301 calculates thelogical OR of the covered background area specified by the component 1,the covered background area specified by the component 2, and thecovered background area specified by the component 3, and sets thecovered background area calculated by logical OR as area information.

[0767] In step S1307, the logical OR processor 1301 determines thelogical OR of the uncovered background area specified by the component1, the uncovered background area specified by the component 2, and theuncovered background area specified by the component 3, and sets theuncovered background area calculated by logical OR as area information.The logical OR processor 1301 outputs the area information in which theforeground area, the background area, the covered background area, andthe uncovered background area are set, and then the processing isterminated.

[0768] As discussed above, the image processing apparatus configured asshown in FIG. 92 specifies an area for each component of the componentsignal and determines the logical OR of the specified areas in order togenerate final area information. The image processing apparatusconfigured as shown in FIG. 92 is able to thoroughly output the areainformation in which the foreground area, the background area, thecovered background area, and the uncovered background area arespecified.

[0769]FIG. 95 shows another embodiment of the image processing apparatusfor generating area information on the basis of an input image which isinput as component signals.

[0770] The same elements as those shown in FIG. 92 are designated withlike reference numerals, and an explanation thereof is thus omitted.

[0771] Based on the area information supplied from the area specifyingunit 103-1, the area information 2 supplied from the area specifyingunit 103-2, and the area information 3 supplied from the area specifyingunit 103-3, a logical AND processor 1321 calculates the logical AND ofthe foreground area indicated by the area information 1, the foregroundarea indicated by the area information 2, the foreground area indicatedby the area information 3, and generates area information in which theforeground area calculated by logical AND is set. Based on the areainformation supplied from the area specifying unit 103-1, the areainformation 2 supplied from the area specifying unit 103-2, and the areainformation 3 supplied from the area specifying unit 103-3, the logicalAND processor 1321 calculates the logical AND of the background areaindicated by the area information 1, the background area indicated bythe area information 2, the background area indicated by the areainformation 3, and generates area information in which the backgroundarea calculated by logical AND is set.

[0772] Based on the area information supplied from the area specifyingunit 103-1, the area information 2 supplied from the area specifyingunit 103-2, and the area information 3 supplied from the area specifyingunit 103-3, the logical AND processor 1321 calculates the logical AND ofthe covered background area indicated by the area information 1, thecovered background area indicated by the area information 2, the coveredbackground area indicated by the area information 3, and generates areainformation in which the covered background area calculated by logicalAND is set. Based on the area information supplied from the areaspecifying unit 103-1, the area information 2 supplied from the areaspecifying unit 103-2, and the area information 3 supplied from the areaspecifying unit 103-3, the logical AND processor 1321 calculates thelogical AND of the uncovered background area indicated by the areainformation 1, the uncovered background area indicated by the areainformation 2, the uncovered background area indicated by the areainformation 3, and generates area information in which the uncoveredbackground area calculated by logical AND is set.

[0773] The logical AND processor 1321 outputs the area information inwhich the foreground area, the background area, the covered backgroundarea, and the uncovered background area are set.

[0774] Next, referring to the flowchart in FIG. 96, a description willnow be given of area determining processing using components, performedby the image processing apparatus configured as shown in FIG. 95.

[0775] In step S1321, the area specifying unit 103-1 executes areaspecifying processing on the basis of the component 1 in order togenerate area information 1 and supplies the generated area information1 to the logical AND processor 1321. The processing of step S1321 is thesame as the processing of step S11, and a detailed explanation thereofis thus omitted.

[0776] In step S1322, the area specifying unit 103-2 executes areaspecifying processing on the basis of the component 2 in order togenerate area information 2 and supplies the generated area information2 to the logical AND processor 1321. The processing of step S1322 is thesame as the processing of step S11, and a detailed explanation thereofis thus omitted.

[0777] In step S1323, the area specifying unit 103-3 executes areaspecifying processing on the basis of the component 3 in order togenerate area information 3 and supplies the generated area information3 to the logical AND processor 1321. The processing of step S1323 is thesame as the processing of step S11, and a detailed explanation thereofis thus omitted.

[0778] In step S1324, the logical AND processor 1321 determines thelogical AND of the foreground area specified by the component 1, theforeground area specified by the component 2, and the foreground areaspecified by the component 3, and sets the foreground area calculated bylogical AND as area information.

[0779] In step S1325, the logical AND processor 1321 determines thelogical AND of the background area specified by the component 1, thebackground area specified by the component 2, and the background areaspecified by the component 3, and sets the background area calculated bylogical AND as area information.

[0780] In step S1326, the logical AND processor 1321 determines thelogical AND of the covered background area specified by the component 1,the covered background area specified by the component 2, and thecovered background area specified by the component 3, and sets thecovered background area calculated by logical AND as area information.

[0781] In step S1327, the logical AND processor 1321 determines thelogical AND of the uncovered background area specified by the component1, the uncovered background area specified by the component 2, and theuncovered background area specified by the component 3, and sets theuncovered background area calculated by logical AND as area information.The logical AND processor 1321 outputs the area information in which theforeground area, the background area, the covered background area, andthe uncovered background area are set, and the processing is thenterminated.

[0782] As discussed above, the image processing apparatus configured asshown in FIG. 95 specifies an area for each component, and determinesthe logical AND of the specified areas in order to generate final areainformation. The image processing apparatus configured as shown in FIG.95 is able to output area information having a small amount of errors.

[0783]FIG. 97 shows still another embodiment of the image processingapparatus for generating area information on the basis of an input imagewhich is input as component signals.

[0784] An area specifying unit 1331 adds, for each pixel, the component1, the component 2, and the component 3, contained in the inputcomponent signals, and determines whether the pixel belongs to a movingarea or a stationary area on the basis of the component 1, the component2, and the component 3, which are added for each pixel. The areaspecifying unit 1331 generates area information on the basis of theresult of the determination of the moving area or the stationary area,and outputs the generated area information.

[0785] Referring to FIGS. 98 to 102, the processing of the areaspecifying unit 1331 will now be described.

[0786] As shown in FIG. 98, from the viewpoint of statisticalcharacteristics of images, the time correlation is stronger than thespace correlation in the stationary area of the image. Furthermore, inthe moving area of the image, conversely, the space correlation isstronger than the time correlation.

[0787] When the space correlation for the determination of the movingarea or the stationary area is to be calculated, as shown in FIG. 99,for example, with regard to the pixels of a block of 5×5 pixels with thedesignated pixel being the center, the area specifying unit 1331calculates the differences of the pixel values of the adjacent pixels,and calculates the total sum of the absolute values of the calculateddifferences. The area specifying unit 1331 calculates the correlationvalue corresponding to the space correlation by dividing the total sumof the absolute values of the calculated differences by the number ofthe differences.

[0788] For example, when the designated pixel is x33 and the pixelvalues of the pixels contained in the block are x11 to x55, the areaspecifying unit 1331 calculates the sum of the absolute values thedifferences, which is the total sum of the absolute values of thedifferences of the pixel values of the adjacent pixels, shown inequation (119): $\begin{matrix}\begin{matrix}{\begin{matrix}{{{Sum}\quad {of}}\quad} \\{\quad {absolute}{\quad \quad}} \\{{{values}\quad {of}}\quad} \\{\quad {differences}}\end{matrix} = {{\left( {{x11} - {x12}} \right)} + {\left( {{x12} - {x13}} \right)} + {\left( {{x13} - {x14}} \right)} + \ldots +}} \\{{{\left( {{x52} - {x53}} \right)} + {\left( {{x53} - {x54}} \right)} + {\left( {{x54} - {x55}} \right)} +}} \\{{{\left( {{x11} - {x21}} \right)} + {\left( {{x21} - {x31}} \right)} + {\left( {{x31} - {x41}} \right)} + \ldots +}} \\{{{\left( {{x25} - {x35}} \right)} + {\left( {{x35} - {x45}} \right)} + {\left( {{x45} - {x55}} \right)} +}}\end{matrix} & (119)\end{matrix}$

[0789] The area specifying unit 1331 calculates the correlation valuecorresponding to the space correlation by dividing the sum of theabsolute values the differences by the difference number, i.e., 40.

[0790] When the time correlation for determining whether or not it is amoving area or a stationary area is to be calculated, the areaspecifying unit 1331 calculates the difference between the pixel valueof the designated pixel and the pixel value of the pixel at thecorresponding position in the previous frame, and also calculates thedifference between the pixel value of the designated pixel and the pixelvalue of the pixel at the corresponding position in the subsequentframe. In order to prevent the area from being determined to be wide inthe boundary as a result of using a difference indicating a strongercorrelation, the area specifying unit 1331 selects either the differencewith respect to the pixel value of the pixel at the correspondingposition in the previous frame or the difference with respect to thepixel value of the pixel at the corresponding position in the previousframe, which is smaller.

[0791] The area specifying unit 1331 calculates the total sum of theabsolute values of the selected differences. The area specifying unit1331 calculates the correlation value corresponding to the timecorrelation by dividing the total sum of the absolute values of thecalculated differences by the number of differences.

[0792] For example, when the designated frame is frame #n, as shown inFIG. 100, the area specifying unit 1331 calculates the differencesbetween respective pixels of a block of 3×3 pixels, for example, withthe designated pixel x22 being the center.

[0793] The area specifying unit 1331 calculates the difference betweenthe pixel value x11 of the pixel in frame #n and the pixel value x11 ofthe pixel at the corresponding position in frame #n−1, and thedifference between the pixel value x11 of the pixel in frame #n and thepixel value x11 of the pixel at the corresponding position in frame#n+1. The area specifying unit 1331 selects either the difference withrespect to the pixel value x11 of the pixel at the correspondingposition in frame #n−1 or the difference with respect to the pixel valuex11 of the pixel at the corresponding position in frame #n+1, which issmaller.

[0794] Similarly, the area specifying unit 1331 calculates thedifference between each of the pixel values x12 to x33 of the pixels inframe #n and each of the pixel values x12 to x33 of the pixels at thecorresponding positions in frame #n−1, and the difference between eachof the pixel values x12 to x33 of the pixels in frame #n and each of thepixel values x12 to x33 of the pixels at the corresponding positions inframe #n+1. The area specifying unit 1331 selects either thecorresponding difference with each of the pixel values x12 to x33 of thepixels at the corresponding positions in frame #n−1 or the correspondingdifference with each of the pixel values x12 to x33 of the pixels at thecorresponding positions in frame #n+1, which is smaller.

[0795] The area specifying unit 1331 determines the total sum of theabsolute values of the selected nine differences. The area specifyingunit 1331 calculates the correlation value corresponding to the timecorrelation by dividing the total sum of the absolute values of thecalculated differences by the number of differences, i.e., 9.

[0796] Furthermore, the area specifying unit 1331 may calculate the timecorrelation and the space correlation for each pixel. Furthermore, thearea specifying unit 1331 may calculate, in addition to the sum of theabsolute values of the differences, values indicating othercorrelations, such as the sum of squares of differences, as valuesindicating the time correlation and the space correlation.

[0797] The number of pixels contained in the block for which the timecorrelation and the space correlation are to be calculated does notlimit the present invention.

[0798]FIG. 101 illustrates a time correlation and a space correlation ina stationary area. For example, in the stationary area, since the imageis stationary, the pixel value J of the designated pixel is the same asthe pixel value J of the pixel at the corresponding position. Therefore,in the stationary area, the time correlation is very strong.

[0799] In contrast, the pixel adjacent to the designated pixel has apixel value I or a pixel value K differing from the pixel value J.

[0800] As discussed above, in the stationary area, the space correlationis weaker than the time correlation.

[0801]FIG. 102 illustrates a time correlation and a space correlation ina moving area. For example, of the components I/4, J/4, K/4, and L/4 ofthe image, contained in the designated pixel in the moving area, thecomponents I/4, J/4, and K/4 of the image are contained in the adjacentpixels, and the components J/4, K/4, and L/4 of the image are containedin other adjacent pixels. Therefore, in the moving area, the spacecorrelation is strong.

[0802] In contrast, the pixel of the adjacent frame, corresponding tothe designated pixel, contains different image components.

[0803] As discussed above, in the moving area, the time correlation isweaker than the space correlation.

[0804] As is seen from the above description, by determining the spacecorrelation and the time correlation and by comparing the spacecorrelation with the time correlation, the area specifying unit 1331 isable to know whether the designated pixel is contained in the movingarea or in the stationary area.

[0805] The area specifying unit 1331 performs the determination of themoving area or the stationary area with respect to all the pixelscontained in the frame.

[0806] Based on the determination result of the moving area or thestationary area, the area specifying unit 1331 determines to which oneof the foreground area, the background area, the covered backgroundarea, and the uncovered background area each pixel belongs, andgenerates area information.

[0807] For example, when the same area as the foreground area isdetermined to be a moving area, the area specifying unit 1331 makes adetermination as to the foreground area, the background area, thecovered background area, and the uncovered background area for eachpixel in the following manner. When the designated pixel in frame #n isdetermined to be moving, the area specifying unit 1331 determines thatthe designated pixel in frame #n belongs to the foreground area.

[0808] When the designated pixel in frame #n is determined to bestationary, and when the pixel in frame #n+1 located at the sameposition as the position of the designated pixel in frame #n in theimage, is determined to be moving, the area specifying unit 1331determines that the designated pixel in frame #n belongs to the coveredbackground area.

[0809] When the designated pixel in frame #n is determined to bestationary, and when the pixel in frame #n−1 located at the sameposition as the position of the designated pixel in frame #n in theimage, is determined to be moving, the area specifying unit 1331determines that the designated pixel in frame #n belongs to theuncovered background area.

[0810] The area specifying unit 1331 determines that the pixel which isdetermined to be stationary and which does not belong to the coveredbackground area or the uncovered background area belongs to thebackground area.

[0811] When the area formed of the background area and the mixed area isdetermined to be a moving area, the area specifying unit 1331 makes adetermination as to the foreground area, the background area, thecovered background area, and the uncovered background area for eachpixel by referring to the stationary area of the adjacent frame.

[0812] A description will now be given of an example of processingresults with reference to FIGS. 103 to 106.

[0813]FIG. 103 shows an example of an input image corresponding to thecomponent 1, the component 2, and the component 3, which are input tothe area specifying unit 1331.

[0814]FIG. 104 shows results in which the area specifying unit 1331makes a determination as to the moving area or the stationary area foreach pixel in such a manner as to correspond to the input image shown inFIG. 103. In FIG. 104, white indicates a moving area, and blackindicates a stationary area. It can be seen that, although there areincorrectly determined portions, the moving area or the stationary areais nearly specified as a whole.

[0815]FIG. 105 shows results in which the area specifying unit 1331 hasmade the determination of the moving area or the stationary area byusing a block of 15×15 pixels as units in such a manner as to correspondto the input image shown in FIG. 103. In FIG. 105, white indicates amoving area, and black indicates a stationary area. It can be seen thatthe moving area or the stationary area is specified nearly accurately.

[0816]FIG. 106 shows results in which the area specifying unit 1331 hasmade the determination of the foreground area, the background area, thecovered background area, and the uncovered background area on the basisof the results of the determination of the moving area or the stationaryarea shown in FIG. 105. In FIG. 106, reference letter A indicates abackground area. Reference letter B indicates an uncovered backgroundarea. Reference letter C indicates a foreground area. Reference letter Dindicates a covered background area.

[0817] It can be seen that, since the area is determined based on thealmost accurate specification of the moving area or the stationary area,the foreground area, the background area, the covered background area,and the uncovered background area can almost be specified accurately.

[0818] The area specifying processing using components, performed by thearea specifying unit 331, is described below with reference to theflowchart of FIG. 107.

[0819] In step S1331, the area specifying unit 1331 calculates the totalsum of the components for each pixel.

[0820] In step S1332, the area specifying unit 1331 determines the spacecorrelation by using, for example, a block composed of a predeterminednumber of pixels as units. In step S1333, the area specifying unit 1331determines the time correlation by using, for example, a block composedof a predetermined number of pixels as units.

[0821] In step S1334, the area specifying unit 1331 compares the spacecorrelation with the time correlation for each pixel in order todetermine whether or not the time correlation is stronger than the spacecorrelation. When it is determined that the time correlation is strongerthan the space correlation, the process proceeds to step S1335, wherethe area specifying unit 1331 sets the designated pixel as being in astationary area, and then the process proceeds to step S1337.

[0822] When it is determined in step S1334 that the time correlation isnot stronger than the space correlation, since the space correlation isstronger than the time correlation, the process proceeds to step S1336,where the area specifying unit 1331 sets the designated pixel as beingin a moving area, and then the process proceeds to step S1337.

[0823] Processings of steps S1334 through S1336 are performed from eachof all the pixels within the frame.

[0824] In step S1337, based on the stationary or moving setting, thearea specifying unit 1331 makes a determination as to the foregroundarea, the background area, the covered background area, or the uncoveredbackground area, and then the processing is terminated.

[0825] For example, when it is determined in step S1337 that thedesignated pixel in frame #n is determined to be moving, the areaspecifying unit 1331 determines that the designated pixel in frame #nbelongs to the foreground area.

[0826] When it is determined that the designated pixel in frame #n isstationary, and when it is determined that the pixel in frame #n+1located at the same position as the position of the designated pixel inframe #n in the image is moving, the area specifying unit 1331determines that the designated pixel in frame #n belongs to the coveredbackground area.

[0827] When it is determined that the designated pixel in frame #n isstationary, and when it is determined that the pixel in frame #n−1located at the same position as the position of the designated pixel inframe #n in the image is moving, the area specifying unit 1331determines that the designated pixel in frame #n belongs to theuncovered background area.

[0828] The area specifying unit 1331 determines that the pixel which isdetermined to be stationary and which does not belong to the coveredbackground area or the uncovered background area belongs to thebackground area.

[0829] As discussed above, based on the components, the area specifyingunit 1331 shown in FIG. 97 is able to specify the foreground area, thebackground area, the covered background area, or the uncoveredbackground area.

[0830]FIG. 108 shows an embodiment of an image processing apparatus forcalculating the mixture ratio α on the basis of input images and areainformation, which are input as component signals.

[0831] A mixture-ratio calculator 104-1 calculates a mixture ratio 1 onthe basis of the area information and the component 1, and supplies thecalculated mixture ratio 1 to an averaging processor 1401. Themixture-ratio calculator 104-1 has the same configuration as that of theestimated-mixture-ratio calculator 104, and an explanation thereof isthus omitted.

[0832] A mixture-ratio calculator 104-2 calculates a mixture ratio 2 onthe basis of the area information and the component 2, and supplies thecalculated mixture ratio 2 to the averaging processor 1401. Themixture-ratio calculator 104-2 has the same configuration as that of theestimated-mixture-ratio calculator 104, and an explanation thereof isthus omitted.

[0833] A mixture-ratio calculator 104-3 calculates a mixture ratio 3 onthe basis of the area information and the component 3, and supplies thecalculated mixture ratio 3 to the averaging processor 1401. Themixture-ratio calculator 104-3 has the same configuration as that of theestimated-mixture-ratio calculator 104, and an explanation thereof isthus omitted.

[0834] The averaging processor 1401 calculates the average value of themixture ratio 1 supplied from the mixture-ratio calculator 104-1, themixture ratio 2 supplied from the mixture-ratio calculator 104-2, andthe mixture ratio 3 supplied from the mixture-ratio calculator 104-3,and outputs the calculated average value as the mixture ratio α.

[0835] As indicated by “A” in FIG. 109, the mixture ratio αcorresponding to the luminance value Y which is the component 1, themixture ratio α corresponding to the color difference U which is thecomponent 2, and the mixture ratio α corresponding to the colordifference V which is the component 3 in a predetermined pixel are thesame. The image processing apparatus configured as shown in FIG. 108 isable to calculate the mixture ratio α more accurately by calculating themixture ratio α using the component 1, the component 2, and thecomponent 3.

[0836] Referring to the flowchart of FIG. 110, a description is given ofthe processing for calculating the mixture ratio α using componentsignals, performed by the image processing apparatus configured as shownin FIG. 108.

[0837] In step S1401, the mixture-ratio calculator 104-1 calculates themixture ratio 1 on the basis of the area information and the component1. The mixture-ratio calculator 104-1 supplies the calculated mixtureratio 1 to the averaging processor 1401. The processing of step S1401 issimilar to that of step S12, and a detailed explanation thereof is thusomitted.

[0838] In step S1402, the mixture-ratio calculator 104-2 calculates themixture ratio 2 on the basis of the area information and the component2. The mixture-ratio calculator 104-2 supplies the calculated mixtureratio 2 to the averaging processor 1401. The processing of step S1402 issimilar to that of step S12, and a detailed explanation thereof is thusomitted.

[0839] In step S1403, the mixture-ratio calculator 104-3 calculates themixture ratio 3 on the basis of the area information and the component3. The mixture-ratio calculator 104-3 supplies the calculated mixtureratio 3 to the averaging processor 1401. The processing of step S1403 issimilar to that of step S12, and a detailed explanation thereof is thusomitted.

[0840] In step S1404, the averaging processor 1401 calculates theaverage value of the mixture ratio 1 based on the component 1, themixture ratio 2 based on the component 2, and the mixture ratio 3 basedon the component 3, and outputs the calculated average value as themixture ratio α. The processing is then terminated.

[0841] As discussed above, the image processing apparatus configured asshown in FIG. 108 calculates the mixture ratio for each component, andcalculates the average value of the calculated mixture ratios in orderto generate the final mixture ratio α. The image processing apparatusconfigured as shown in FIG. 108 is able to calculate the mixture ratio αin which an influence due to an error which occurs in one component isreduced.

[0842]FIG. 111 shows another embodiment of an image processing apparatusfor calculating the mixture ratio α on the basis of input images andarea information, which are input as component signals. The sameelements as those shown in FIG. 108 are designated with like referencenumerals, and an explanation thereof is thus omitted.

[0843] A majority processor 1411 classifies the mixture ratio 1 suppliedfrom the mixture-ratio calculator 104-1, the mixture ratio 2 suppliedfrom the mixture-ratio calculator 104-2, and the mixture ratio 3supplied from the mixture ratio calculator 104-3 at a predeterminedinterval, and determines the frequency corresponding to therepresentative value of the interval. The majority processor 1411determines the mixture ratio α on the basis of the frequencycorresponding to the representative value and outputs the determinedmixture ratio α.

[0844] For example, when the width of the interval is 0.1 and therepresentative values are 0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8,0.9, and 1.0, and when the mixture ratio 1 is 0.12, the mixture ratio 2is 0.13, and the mixture ratio 3 is 0.41, the majority processor 1411determines the frequency of the representative value 0.1 to be 2 anddetermines the frequency of the representative value 0.4 to be 1. Themajority processor 1411 sets the representative value 0.1 correspondingto the largest frequency 2 to the mixture ratio α.

[0845] Referring to the flowchart of FIG. 112, a description is given ofthe processing for calculating the mixture ratio α, using components,performed by the area specifying unit 331 configured as shown in FIG.111.

[0846] The processings of step S1411 through step S1413 are similar tothose of step S1401 through step S1403, respectively, and an explanationthereof is thus omitted.

[0847] In step S1414, the majority processor 1411 classifies the mixtureratio of each component at a predetermined interval, and determines thefrequency.

[0848] In step S1415, the majority processor 1411 determines the mixtureratio α on the basis of the frequency, and then the processing isterminated.

[0849] As discussed above, the image processing apparatus configured asshown in FIG. 111 calculates the mixture ratio for each component, andgenerates the final mixture ratio α on the basis of the frequency of thecalculated mixture ratio. The image processing apparatus configured asshown in FIG. 111 is able to calculate a highly reliable mixture ratio αin which an influence of a mixture ratio whose value differs greatly dueto an error of a single component is eliminated.

[0850]FIG. 113 shows still another embodiment of the image processingapparatus for calculating the mixture ratio α on the basis of inputimages and the area information, which are inputs as component signals.

[0851] A mixture-ratio calculator 1421 adds, for each pixel, the pixelvalues of the input component 1, component 2, and component 3,calculates the mixture ratio α on the basis of the added pixel values ofthe component 1, the component 2, and the component 3 and the areainformation, and outputs the calculated mixture ratio α.

[0852]FIG. 114 is a block diagram illustrating the configuration of themixture-ratio calculator 1421. An adder 1431 adds, for each pixel, thepixel values of the input component 1, component 2, and component 3, andsupplies the added value to the estimated-mixture-ratio processor 401and the estimated-mixture-ratio processor 402.

[0853] Based on the value in which the pixel values of the component 1,the component 2, and the component 3 are added for each pixel, theestimated-mixture-ratio processor 401 calculates the estimated mixtureratio for each pixel by calculations corresponding to a model in thecovered background area, and supplies the calculated estimated mixtureratio to the mixture-ratio determining portion 403.

[0854] Equation (21) showing the mixture ratio α of the pixel belongingto the covered background area can be expressed by equation (120)through equation (122) for each component:

αY≈(CY−NY)/(PY−NY)  (120)

αU≈(CU−NU)/(PU−NU)  (121)

αV≈(CV−NV)/(PV−NV)  (122)

[0855] CY denotes a pixel value of frame #n in the component 1, which isthe luminance value Y. NY denotes a pixel value of frame #n+1 which issubsequent to frame #n in the component 1. PY denotes a pixel value offrame #n−1 which is previous to frame #n in the component 1.

[0856] CU denotes a pixel value of frame #n in the component 2, which isthe color difference U. NU denotes a pixel value of frame #n+1 which issubsequent to frame #n in the component 2. PU denotes a pixel value offrame #n−1 which is previous to frame #n in the component 2.

[0857] CV denotes a pixel value of frame #n in the component 3, which isthe color difference V. NV denotes a pixel value of frame #n+1 which issubsequent to frame #n in the component 3. PV denotes a pixel value offrame #n−1 which is previous to frame #n in the component 3.

[0858] Since the mixture ratio α to be calculated is the same value inthe components 1 to 3, equation (123) holds:

αY=αU=αV  (123)

[0859] Equation (124) can be derived from equations (120) to (123).

(CY−NY)−(PY−NY)=(CU−NU)/(PU−NU)=(CV−NV)/(PV−NV)  (124)

[0860] Furthermore, equation (125) for calculating the mixture ratio αcan be derived from equation (124).

α=((CY+CU+CV)−(NY+NU+NV))/((PY+PU+PV)−(NY+NU+NV))  (125)

[0861] As discussed above, the mixture ratio α can be calculated basedon the value in which the pixel values of the component 1, the component2, and the component 3 are added.

[0862] The estimated-mixture-ratio processor 402 calculates an estimatedmixture ratio for each pixel by calculations corresponding to a model ofan uncovered background area based on the value in which the pixelvalues of the component 1, the component 2, and the component 3 areadded for each pixel, and supplies the calculated estimated mixtureratio to the mixture-ratio determining portion 403.

[0863] The mixture-ratio determining portion 403 sets the mixture ratioα to 0 when the target pixel belongs to the foreground area, and setsthe mixture ratio α to 1 when the target pixel belongs to the backgroundarea. When the target pixel belongs to the covered background area, themixture-ratio determining portion 403 sets the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 401 as the mixtureratio α. When the target pixel belongs to the uncovered background area,the mixture-ratio determining portion 403 sets the estimated mixtureratio supplied from the estimated-mixture-ratio processor 402 as themixture ratio α. The mixture-ratio determining portion 403 outputs themixture ratio α which has been set based on the area information.

[0864] As discussed above, by using the component 1, the component 2,and the component 3, the mixture-ratio calculator 1421 is able tocalculate the mixture ratio α having a higher accuracy than the mixtureratio α calculated based on a single signal.

[0865] Next, referring to the flowchart of FIG. 115, a description isgiven of the processing for calculating the mixture ratio α based on theinput image and the area information, which are input as componentsignals, performed by the image processing apparatus of FIG. 113. Instep S1421, an adder 1431 adds up the pixel values of the inputcomponent 1, component 2, and component 3 for each pixel. The adder 1431supplies the added-up pixel value to the estimated-mixture-ratioprocessor 401 and the estimated-mixture-ratio processor 402.

[0866] In step S1422, based on the added-up pixel value, theestimated-mixture-ratio processor 401 calculates the estimated mixtureratio for each pixel by a calculation corresponding to a model of acovered background area. The estimated-mixture-ratio processor 401supplies the calculated estimated mixture ratio to the mixture-ratiodetermining portion 403. Details of the processing of step S1422 aresimilar to those of the processing of step S402, and an explanationthereof is thus omitted.

[0867] In step S1423, based on the added-up pixel value, theestimated-mixture-ratio processor 402 calculates the estimated mixtureratio for each pixel by calculations corresponding to a model of theuncovered background area. The estimated-mixture-ratio processor 402supplies the calculated estimated mixture ratio to the mixture-ratiodetermining portion 403. Details of the processing of step S1423 aresimilar to those of the processing of step S403, and an explanationthereof is thus omitted.

[0868] In step S1424, the mixture-ratio calculator 104 determineswhether the mixture ratios have been estimated for the whole frame. Ifit is determined that the mixture ratios have not yet been estimated forthe whole frame, the process returns to step S1422, and the processingfor estimating the mixture ratio for the subsequent pixel is repeated.

[0869] If it is determined in step S1424 that the mixture ratios havebeen estimated for the whole frame, the process proceeds to step S1425.In step S1425, the mixture-ratio determining portion 403 sets themixture ratio α based on the area information indicating to which of theforeground area, the background area, the covered background area, orthe uncovered background area the pixel belongs. The mixture-ratiodetermining portion 403 sets the mixture ratio α to 0 when the targetpixel belongs to the foreground area, and sets the mixture ratio α to 1when the target pixel belongs to the background area. When the targetpixel belongs to the covered background area, the mixture-ratiodetermining portion 403 sets the estimated mixture ratio supplied fromthe estimated-mixture-ratio processor 401 as the mixture ratio α. Whenthe target pixel belongs to the uncovered background area, themixture-ratio determining portion 403 sets the estimated mixture ratiosupplied from the estimated-mixture-ratio processor 402 as the mixtureratio α. The processing is then terminated.

[0870] As discussed above, the image processing apparatus configured asshown in FIG. 113 is able to calculate the mixture ratio α, which is afeature quantity, corresponding to each pixel, with higher accuracy onthe basis of the area information, the component 1, the component 2, andthe component 3.

[0871] The embodiment has been discussed above by setting the mixtureratio α to the ratio of the background components contained in the pixelvalues. However, the mixture ratio α may be set to the ratio of theforeground components contained in the pixel values.

[0872] The embodiment has been discussed above by setting the movingdirection of the foreground object to the direction from the left to theright. However, the moving direction is not restricted to theabove-described direction. In the above description, a real-space imagehaving a three-dimensional space and time axis information is projectedonto a time space having a two-dimensional space and time axisinformation by using a video camera. However, the present invention isnot restricted to this example, and can be applied to the followingcase. When a greater amount of first information in one-dimensionalspace is projected onto a smaller amount of second information in atwo-dimensional space, distortion generated by the projection can becorrected, significant information can be extracted, or a more naturalimage can be synthesized.

[0873] The sensor is not restricted to a CCD, and may be another type ofsensor, such as a solid-state image-capturing device, for example, aCMOS (Complementary Metal Oxide Semiconductor) a BBD (Bucket BrigadeDevice), a CID (Charge Injection Device), or a CPD (Charge PrimingDevice). Also, the sensor does not have to be a sensor in whichdetection devices are arranged in a matrix, and may be a sensor in whichdetection devices are arranged in one line.

[0874] A recording medium in which a program for performing the signalprocessing of the present invention is recorded may be formed of apackaged medium in which the program is recorded, which is distributedfor providing the program to a user separately from the computer, asshown in FIG. 1, such as the magnetic disk 51 (including a floppy(registered trade name) disk), the optical disc 52 (CD-ROM (CompactDisc-Read Only Memory) and a DVD (Digital Versatile Disc)), themagneto-optical disk 53 (including MD (Mini-Disc) (registered tradename)), or the semiconductor memory 54. The recording medium may also beformed of the ROM 22 or a hard disk contained in the storage unit 28 inwhich the program is recorded, such recording medium being provided tothe user while being prestored in the computer.

[0875] The steps forming the program recorded in a recording medium maybe executed chronologically according to the orders described in thespecification. However, they do not have to be executed in a time-seriesmanner, and they may be executed concurrently or individually.

INDUSTRIAL APPLICABILITY

[0876] According to a first aspect of the invention, the mixture stateof images can be recognized.

[0877] According to a second aspect of the invention, the mixture stateof images can be recognized.

1. An image processing apparatus for processing image data which isformed of a predetermined number of pixel data, having a plurality oftypes of components at the same pixel position, obtained by animage-capturing device including a predetermined number of pixels, thepixels having a time integrating function, said image processingapparatus comprising: area specifying means for specifying, incorrespondence with said image data, a mixed area in which foregroundobject components which form a foreground object and background objectcomponents which form a background object are mixed; and mixture-ratiodetection means for detecting, in correspondence with said image data,the mixture ratio indicating the ratio of the mixture of said foregroundobject components to the mixture of said background object components ina mixed area in which said foreground object components and saidbackground object components are mixed, wherein at least one of saidarea specifying means and said mixture-ratio detection means performsimage processing on the basis of said plurality of types of components.2. An image processing apparatus according to claim 1, wherein said areaspecifying means comprises: component mixed-area detection means fordetecting said mixed area for each of said plurality of types ofcomponents and for outputting the detection result corresponding toindividual components as component mixed-area information; andmixed-area specifying means for specifying said mixed area correspondingto said image data on the basis of the detection result of said mixedarea corresponding to said plurality of types of components detected bysaid component mixed-area detection means.
 3. An image processingapparatus according to claim 1, wherein said area specifying meanscomprises: space-correlation-value calculation means for calculating aspace correlation value indicating a correlation between designatedpixel data corresponding to a designated pixel of a designated frame ofsaid image data and pixel data of a space neighboring pixel positionedin the neighborhood of said designated pixel in the spatial direction onthe basis of said plurality of types of components corresponding to saiddesignated pixel; time-correlation-value calculation means forcalculating a time correlation value indicating a correlation betweensaid designated pixel data and pixel data of a time neighboring pixelpositioned in the neighborhood of said designated pixel in the timedirection on the basis of said plurality of types of componentscorresponding to said designated pixel; and foreground area specifyingmeans for specifying a foreground area formed of only said foregroundobject components on the basis of said space correlation value and saidtime correlation value corresponding to said designated pixel.
 4. Animage processing apparatus according to claim 3, wherein said areaspecifying means further comprises mixed-area specifying means forspecifying said mixed area on the basis of said foreground area of saiddesignated frame and said foreground area of a neighboring frame in theneighborhood of said designated frame.
 5. An image processing apparatusaccording to claim 1, wherein said mixture-ratio detection meanscomprises: component mixture-ratio detection means for detecting themixture ratio for each of said plurality of types of components; andcomponent integrated mixture-ratio detection means for detecting themixture ratio corresponding to said image data by integrating thedetection results of the mixture ratios corresponding to said pluralityof types of components detected by said component mixture-ratiodetection means.
 6. An image processing apparatus according to claim 1,wherein said mixture-ratio detection means comprises: integration meansfor integrating the pixel values of said plurality of types ofcomponents for each pixel and for outputting the value as integrateddata; and integrated data mixture-ratio detection means for detectingthe mixture ratio corresponding to said image data on the basis of saidintegrated data.
 7. An image processing apparatus according to claim 6,wherein said integration means adds said pixel values of said pluralityof types of components for each pixel and outputs the added result assaid integrated data.
 8. An image processing method for use with animage processing apparatus for processing image data which is formed ofa predetermined number of pixel data, having a plurality of types ofcomponents at the same pixel position, obtained by an image-capturingdevice including a predetermined number of pixels, the pixels having atime integrating function, said image processing method comprising: anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed; a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed; and an output control step of controlling theoutput of said detected mixture ratio, wherein at least one of said areaspecifying step and said mixture-ratio detection step performs imageprocessing on the basis of said plurality of types of components.
 9. Animage processing method according to claim 8, wherein said areaspecifying step comprises: a component mixed-area detection step ofdetecting said mixed area for each of said plurality of types ofcomponents and for outputting the detection result corresponding toindividual components as component mixed-area information; and amixed-area specifying step of specifying said mixed area correspondingto said image data on the basis of the detection result of said mixedarea corresponding to said plurality of types of components detected insaid component mixed-area detection step.
 10. An image processing methodaccording to claim 8, wherein said area specifying step comprises: aspace-correlation-value calculation step of calculating a spacecorrelation value indicating a correlation between designated pixel datacorresponding to a designated pixel of a designated frame of said imagedata and pixel data of a space neighboring pixel positioned in theneighborhood of said designated pixel in the spatial direction on thebasis of said plurality of types of components corresponding to saiddesignated pixel; a time-correlation-value calculation step ofcalculating a time correlation value indicating a correlation betweensaid designated pixel data and pixel data of a time neighboring pixelpositioned in the neighborhood of said designated pixel in the timedirection on the basis of said plurality of types of componentscorresponding to said designated pixel; and a foreground area specifyingstep of specifying a foreground area formed of only said foregroundobject components on the basis of said space correlation value and saidtime correlation value corresponding to said designated pixel.
 11. Animage processing method according to claim 10, wherein said areaspecifying step further comprises a mixed-area specifying step ofspecifying said mixed area on the basis of said foreground area of saiddesignated frame and said foreground area of a neighboring frame in theneighborhood of said designated frame.
 12. An image processing methodaccording to claim 8, wherein said mixture-ratio detection stepcomprises: a component mixture-ratio detection step of detecting themixture ratio for each of said plurality of types of components; and acomponent integrated mixture-ratio detection step of detecting themixture ratio corresponding to said image data by integrating thedetection results of the mixture ratios corresponding to said pluralityof types of components detected in said component mixture-ratiodetection step.
 13. An image processing method according to claim 8,wherein said mixture-ratio detection step comprises: an integration stepof integrating the pixel values of said plurality of types of componentsfor each pixel and for outputting the value as integrated data; and adata mixture-ratio detection step of detecting the mixture ratiocorresponding to said image data on the basis of said integrated data.14. An image processing method according to claim 13, wherein, in saidintegration step, said pixel values of said plurality of types ofcomponents are added for each pixel, and the added result is output assaid integrated data.
 15. A recording medium having recorded thereon acomputer-readable program which is used to process image data which isformed of a predetermined number of pixel data, having a plurality oftypes of components at the same pixel position, obtained by animage-capturing device including a predetermined number of pixels, thepixels having a time integrating function, said program comprising: anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed; a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed; and an output control step of controlling theoutput of said detected mixture ratio, wherein at least one of said areaspecifying step and said mixture-ratio detection step performs imageprocessing on the basis of said plurality of types of components.
 16. Arecording medium according to claim 15, wherein said area specifyingstep comprises: a component mixed-area detection step of detecting saidmixed area for each of said plurality of types of components and foroutputting the detection result corresponding to individual componentsas component mixed-area information; and a mixed-area specifying step ofspecifying said mixed area corresponding to said image data on the basisof the detection result of said mixed area corresponding to saidplurality of types of components detected in said component mixed-areadetection step.
 17. A recording medium according to claim 15, whereinsaid area specifying step comprises: a space-correlation-valuecalculation step of calculating a space correlation value indicating acorrelation between designated pixel data corresponding to a designatedpixel of a designated frame of said image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of said designatedpixel in the spatial direction on the basis of said plurality of typesof components corresponding to said designated pixel; atime-correlation-value calculation step of calculating a timecorrelation value indicating a correlation between said designated pixeldata and pixel data of a time neighboring pixel positioned in theneighborhood of said designated pixel in the time direction on the basisof said plurality of types of components corresponding to saiddesignated pixel; and a foreground area specifying step of specifying aforeground area formed of only said foreground object components on thebasis of said space correlation value and said time correlation valuecorresponding to said designated pixel.
 18. A recording medium accordingto claim 17, wherein said area specifying step further comprises amixed-area specifying step of specifying said mixed area on the basis ofsaid foreground area of said designated frame and said foreground areaof a neighboring frame in the neighborhood of said designated frame. 19.A recording medium according to claim 15, wherein said mixture-ratiodetection step comprises: a component mixture-ratio detection step ofdetecting the mixture ratio for each of said plurality of types ofcomponents; and a component integrated mixture-ratio detection step ofdetecting the mixture ratio corresponding to said image data byintegrating the detection results corresponding to said plurality oftypes of components detected in said component mixture-ratio detectionstep.
 20. A recording medium according to claim 15, wherein saidmixture-ratio detection step comprises: an integration step ofintegrating the pixel values of said plurality of types of componentsfor each pixel and outputting the value as integrated data; and anintegrated data mixture-ratio detection step of detecting the mixtureratio corresponding to said image data on the basis of said integrateddata.
 21. A recording medium according to claim 20, wherein, in saidintegration step, said pixel values of said plurality of types ofcomponents are added for each pixel, and the added result is output assaid integrated data.
 22. A program for enabling a computer to execute,said computer being used to process image data which is formed of apredetermined number of pixel data, having a plurality of types ofcomponents at the same pixel position, obtained by an image-capturingdevice including a predetermined number of pixels, the pixels having atime integrating function: an area specifying step of specifying, incorrespondence with said image data, a mixed area in which foregroundobject components which form a foreground object and background objectcomponents which form a background object are mixed; a mixture-ratiodetection step of detecting, in correspondence with said image data, themixture ratio indicating the ratio of the mixture of said foregroundobject components to the mixture of said background object in a mixedarea in which said foreground object components and said backgroundobject components are mixed; and an output control step of controllingthe output of said detected mixture ratio, wherein, in at least one ofsaid area specifying step and said mixture-ratio detection step, imageprocessing is performed on the basis of said plurality of types ofcomponents.
 23. A program according to claim 22, wherein said areaspecifying step comprises: a component mixed-area detection step ofdetecting said mixed area for each of said plurality of types ofcomponents and for outputting the detection result corresponding toindividual components as component mixed-area information; and amixed-area specifying step of specifying said mixed area correspondingto said image data on the basis of the detection result of said mixedarea corresponding to said plurality of types of components detected insaid component mixed-area detection step.
 24. A program according toclaim 22, wherein said area specifying step comprises: aspace-correlation-value calculation step of calculating a spacecorrelation value indicating a correlation between designated pixel datacorresponding to a designated pixel of a designated frame of said imagedata and pixel data of a space neighboring pixel positioned in theneighborhood of said designated pixel in the spatial direction on thebasis of said plurality of types of components corresponding to saiddesignated pixel; a time-correlation-value calculation step ofcalculating a time correlation value indicating a correlation betweensaid designated pixel data and pixel data of a time neighboring pixelpositioned in the neighborhood of said designated pixel in the timedirection on the basis of said plurality of types of componentscorresponding to said designated pixel; and a foreground area specifyingstep of specifying a foreground area formed of only said foregroundobject components on the basis of said space correlation value and saidtime correlation value corresponding to said designated pixel.
 25. Aprogram according to claim 24, wherein said area specifying step furthercomprises a mixed-area specifying step of specifying said mixed area onthe basis of said foreground area of said designated frame and saidforeground area of a neighboring frame in the neighborhood of saiddesignated frame.
 26. A program according to claim 22, wherein saidmixture-ratio detection step comprises: a component mixture-ratiodetection step of detecting the mixture ratio for each of said pluralityof types of components; and a component integrated mixture-ratiodetection step of detecting the mixture ratio corresponding to saidimage data by integrating the detection results of the mixture ratioscorresponding to said plurality of types of components detected in saidcomponent mixture-ratio detection step.
 27. A program according to claim22, wherein said mixture-ratio detection step comprises: an integrationstep of integrating the pixel values of said plurality of types ofcomponents for each pixel and for outputting the value as integrateddata; and an integrated data mixture-ratio detection step of detectingthe mixture ratio corresponding to said image data on the basis of saidintegrated data.
 28. A program according to claim 27, wherein saidintegration step adds said pixel values of said plurality of types ofcomponents for each pixel and outputs the added result as saidintegrated data.
 29. An image-capturing apparatus comprising:image-capturing means for outputting a subject image captured by animage-capturing device including a predetermined number of pixels, thepixels having a time integrating function, as image data which is formedof a predetermined number of pixel data having a plurality of types ofcomponents at the same pixel position; area specifying means forspecifying, in correspondence with said image data, a mixed area inwhich foreground object components which form a foreground object andbackground object components which form a background object are mixed;and mixture-ratio detection means for detecting, in correspondence withsaid image data, the mixture ratio indicating the ratio of the mixtureof said foreground object components to the mixture of said backgroundobject components in a mixed area in which said foreground objectcomponents and said background object components are mixed, wherein atleast one of said area specifying means and said mixture-ratio detectionmeans performs image processing on the basis of said plurality of typesof components.
 30. An image-capturing apparatus according to claim 29,wherein said area specifying means comprises: component mixed-areadetection means for detecting said mixed area for each of said pluralityof types of components and for outputting the detection resultcorresponding to individual components as component mixed-areainformation; and mixed-area specifying means for specifying said mixedarea corresponding to said image data on the basis of the detectionresult of said mixed area corresponding to said plurality of types ofcomponents detected by said component mixed-area detection means.
 31. Animage-capturing apparatus according to claim 29, wherein said areaspecifying means comprises: space-correlation-value calculation meansfor calculating a space correlation value indicating a correlationbetween designated pixel data corresponding to a designated pixel of adesignated frame of said image data and pixel data of a spaceneighboring pixel positioned in the neighborhood of said designatedpixel in the spatial direction on the basis of said plurality of typesof components corresponding to said designated pixel;time-correlation-value calculation means for calculating a timecorrelation value indicating a correlation between said designated pixeldata and pixel data of a time neighboring pixel positioned in theneighborhood of said designated pixel in the time direction on the basisof said plurality of types of components corresponding to saiddesignated pixel; and foreground area specifying means for specifying aforeground area formed of only said foreground object components on thebasis of said space correlation value and said time correlation valuecorresponding to said designated pixel.
 32. An image-capturing apparatusaccording to claim 31, wherein said area specifying means furthercomprises mixed-area specifying means for specifying said mixed area onthe basis of said foreground area of said designated frame and saidforeground area of a neighboring frame in the neighborhood of saiddesignated frame.
 33. An image-capturing apparatus according to claim29, wherein said mixture-ratio detection means comprises: componentmixture-ratio detection means for detecting the mixture ratio for eachof said plurality of types of components; and component integratedmixture-ratio detection means for detecting the mixture ratio fordetecting the mixture ratio corresponding to said image data byintegrating the detection results of the mixture ratios corresponding tosaid plurality of types of components detected by said componentmixture-ratio detection means.
 34. An image-capturing apparatusaccording to claim 29, wherein said mixture-ratio detection meanscomprises: integration means for integrating the pixel values of saidplurality of types of components for each pixel and for outputting thevalue as integrated data; and integrated data mixture-ratio detectionmeans for detecting the mixture ratio corresponding to said image dataon the basis of said integrated data.
 35. An image-capturing apparatusaccording to claim 34, wherein said integration means adds said pixelvalues of said plurality of types of components for each pixel andoutputs the added result as said integrated data.
 36. An imageprocessing apparatus for processing image data which is formed of apredetermined number of pixel data, having a plurality of types ofcomponents at the same pixel position, obtained by an image-capturingdevice including a predetermined number of pixels, the pixels having atime integrating function, said image processing apparatus comprising:image data obtaining means for obtaining said image data; and processingperforming means for performing, on the basis of said plurality of typesof components of said obtained image data, one of processings of (i) anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed.
 37. An image processing apparatus according toclaim 36, wherein said processing performing means performs, on thebasis of said plurality of types of components of said obtained imagedata, an area specifying step of specifying, in correspondence with saidimage data, a mixed area in which foreground object components whichform a foreground object and background object components which form abackground object are mixed.
 38. An image processing apparatus accordingto claim 36, wherein said processing performing means performs, on thebasis of said plurality of types of components of said obtained imagedata, a mixture-ratio detection step of detecting, in correspondencewith said image data, the mixture ratio indicating the ratio of themixture of said foreground object components to the mixture of saidbackground object components in a mixed area in which said foregroundobject components and said background object components are mixed. 39.An image processing method for processing image data which is formed ofa predetermined number of pixel data, having a plurality of types ofcomponents at the same pixel position, obtained by an image-capturingdevice including a predetermined number of pixels, the pixels having atime integrating function, said image processing method comprising: animage data obtaining step of obtaining said image data; and a processingperforming step of performing, on the basis of said plurality of typesof components of said obtained image data, one of processings of (i) anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed.
 40. An image processing method according to claim39, wherein, in said processing performing step, on the basis of saidplurality of types of components of said obtained image data, an areaspecifying step of specifying, in correspondence with said image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed is performed.
 41. An image processing method according toclaim 39, wherein, in said processing performing step, on the basis ofsaid plurality of types of components of said obtained image data, amixture-ratio detection step of detecting, in correspondence with saidimage data, the mixture ratio indicating the ratio of the mixture ofsaid foreground object components to the mixture of said backgroundobject components in a mixed area in which said foreground objectcomponents and said background object components are mixed is performed.42. A recording medium having recorded thereon a computer-readableprogram which is used to process image data which is formed of apredetermined number of pixel data, having a plurality of types ofcomponents at the same pixel position, obtained by an image-capturingdevice including a predetermined number of pixels, the pixels having atime integrating function, said program comprising: an image dataobtaining step of obtaining said image data; and a processing performingstep of performing, on the basis of said plurality of types ofcomponents of said obtained image data, one of processings of (i) anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed.
 43. A recording medium according to claim 42,wherein, in said processing performing step, on the basis of saidplurality of types of components of said obtained image data, an areaspecifying step of specifying, in correspondence with said image data, amixed area in which foreground object components which form a foregroundobject and background object components which form a background objectare mixed is performed.
 44. A recording medium according to claim 42,wherein, in said processing performing step, on the basis of saidplurality of types of components of said obtained image data, amixture-ratio detection step of detecting, in correspondence with saidimage data, the mixture ratio indicating the mixture of said foregroundobject components to the mixture of said background object components ina mixed area in which said foreground object components and saidbackground object components are mixed is performed.
 45. A program forenabling a computer to execute, said computer being used to processimage data which is formed of a predetermined number of pixel data,having a plurality of types of components at the same pixel position,obtained by an image-capturing device including a predetermined numberof pixels, the pixels having a time integrating function: an image dataobtaining step of obtaining said image data; and a processing performingstep of performing, on the basis of said plurality of types ofcomponents of said obtained image data, one of processings of (i) anarea specifying step of specifying, in correspondence with said imagedata, a mixed area in which foreground object components which form aforeground object and background object components which form abackground object are mixed and (ii) a mixture-ratio detection step ofdetecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed.
 46. A program according to claim 45, wherein, insaid processing performing step, on the basis of said plurality of typesof components of said obtained image data, an area specifying step ofspecifying, in correspondence with said image data, a mixed area inwhich foreground object components which form a foreground object andbackground object components which form a background object are mixed isperformed.
 47. A program according to claim 45, wherein, in saidprocessing performing step, on the basis of said plurality of types ofcomponents of said obtained image data, a mixture-ratio detection stepof detecting, in correspondence with said image data, the mixture ratioindicating the ratio of the mixture of said foreground object componentsto the mixture of said background object components in a mixed area inwhich said foreground object components and said background objectcomponents are mixed is performed.
 48. An image-capturing apparatuscomprising: image-capturing means for outputting a subject imagecaptured by an image-capturing device including a predetermined numberof pixels, the pixels having a time integrating function, as image datawhich is formed of a predetermined number of pixel data having aplurality of types of components at the same pixel position; andprocessing performing means for performing, on the basis of saidplurality of types of components of said image data, one of processingsof (i) an area specifying step of specifying, in correspondence withsaid image data, a mixed area in which foreground object componentswhich form a foreground object and background object components whichform a background object are mixed and (ii) a mixture-ratio detectionstep of detecting, in correspondence with said image data, the mixtureratio indicating the ratio of the mixture of said foreground objectcomponents to the mixture of said background object components in amixed area in which said foreground object components and saidbackground object components are mixed.
 49. An image-capturing apparatusaccording to claim 48, wherein said processing performing meansperforms, on the basis of said plurality of types of components of saidimage data, an area specifying step of specifying, in correspondencewith said image data, a mixed area in which foreground object componentswhich form a foreground object and background object components whichform a background object are mixed.
 50. An image processing apparatusaccording to claim 48, wherein said processing performing meansperforms, on the basis of said plurality of types of components of saidimage data, a mixture-ratio detection step of detecting, incorrespondence with said image data, the mixture ratio indicating theratio of the mixture of said foreground object components to the mixtureof said background object components in a mixed area in which saidforeground object components and said background object components aremixed.